The CookBook Approach To Making Up For Missing Documentation February 26, 2008

The last bunch of people working on the code were a bunch of crack smoking monkeys. It’s landed in your lap like a wet cat and it’s your job to pull it all together, make it work and ship it yesterday. The entire documentation for the project is a patchy user manual and the people responsible who know what it’s supposed to do have moved on after being told that maybe they’d be more suited to something like, well, knitting or demolition.

I’m going to detail a strategy that I’ve used a couple of times now successfully to help fill in the cracks. I figured it’s probably worth writing up so I remember it and can point it out anonymously. Look, it’s on the internet, it must be authoritative. It won’t give you pristine documentation but I’ve found at least, it will give you a basis to build on for the next few releases. If you’ve got nothing and you can’t stop and write it all up (who can?) you’re not going to be able to fix it all in a single release. You can plan to fill it in though and see the light at the end of the tunnel. First of all you’ve got a project wiki right? Something you and the team can write informal documentation on? If you haven’t go download and install somewhere, and come back later. Ok so on to the story.

My grandma had a cookbook. It wasn’t a hard cover printed manual with a picture of an attractive (or not so attractive) chef gracing the cover. It was a behemoth. It had escaped the binder that contained it. It had devoured other cookbooks. When my grandmother had particularly liked something she had liberated it from it’s uniformity in a published cookbook, ripped it out and crammed it in there. There were handwritten recipes in an unfamiliar hand that I’m sure she had brought from distant ancestors. It had food stains on it. Maybe she thought if you couldn’t read it any more at least the food stains would give an indication of the ingredients. An early form of scratch and sniff maybe? Here’s the rub though, it was disorganised and in a lot of case incomplete but she knew it back to front and in between. Based on her knowledge of the world she could use it to build on and serve consistently high quality meals. When someone at dinner said ‘I must get the recipe for this’. She would know exactly where the recipe was, she’d take it out, rewrite it filling in all the skipped steps and then someone else could make something great.

So nice story? but how does this relate to code and documentation? I’m going to suggest you start building a cookbook. This is where the wiki comes in. A cookbook isn’t complete. It doesn’t contain the thousands of variations on macaroni and cheese, but it does usually have a single recipe which captures the intent.

So in this context, what’s a recipe. Well the way I’ve thought about it is a recipe in a application context usually covers a simple functional area. Eg employee mangement. It will have a UI usually for creating, editing, deleting and updating. So it has a datasource which in most cases is hooked up to a db and may encompass a database and maybe some stored procedures. There’s probably some data integrity constraints or some interesting behaviour. It may be related to another recipe, eg organisation hierarchy. It may have some gotchas, deleting a employee only sets an additional field indicating they’re no longer active but doesn’t actually delete them. There’s no hard and fast rule but you need to come up with something people can add to and build on so that you can start filling in what the system does. I usually create a really simple template.

• Functional Area Description – what does it do, who uses it, when do they use it
• UI – Basic sequence, create with this form, delete with this form, this validation sequence is important
• Data – These are the tables that are involved, this uses this specific algorithm
• Gotchas – General things that might trip someone up. The worse the code the more important this is.

For a first release 50-80% coverage is good. You start to build an understanding of the system. Future specifications actually have a basis and documenters, testers, developers can start adding and reading it. You get a bit more time on the next release because some of it’s already documented and you’re not starting from scratch. You start organising it into categories and breaking it down a bit more. You start adding things like schema diagrams, system UML sequences and data integrity documentation, glossaries, table of contents. Eventually you end up with pretty good documentation. I ‘ve found that development actually gets faster, when someone asks or someone new joins the answer is, it’s on the wiki in the cookbook.

This is a strategy I’ve found helpful, it’s not perfect but it’s often a good start and a good conversation a team can have with familiar concepts.

Ill Will : Quantifying Technical Debt February 8, 2008

You can’t say you’ve climbed it until you’ve reached the peak, it doesn’t count unless you ship it. To get to the point where someone pays something for your software, there’s a lot of leeway and like a chess game many small moves that will affect the outcome. You can cut corners, drop features, play with resources, all of which define the history of the application and start adding technical debt. Behind every line of code is debt; wages, premises, capital equipment that someone will eventually pay for.

Technical debt is one of those intangible things that make or breaks an application, sometimes whole companies. Unfortunately it’s one of those things that I think software engineers as a profession don’t do well in conveying the importance and long term effects of. It’s often impossible for outsiders to understand the difference between a quick hack and rigorously tested application as they can appear to be exactly the same thing. Shipping software isn’t a rigorous scientific process and offers a lot of flexibility in getting to and even defining the end point. The call to arms here is that developers must get better at informing users as to costs and importance of technical debt.

Businesses and accountants are already familiar with the fact that the actions of a business can have an affect on the overall success of that business. When a business is acquired there is an allowance for good will. Favourable things the business has done that values the business more than it’s books report. While it can be haggled over, a business with loyal and repeat customers leading to a steady income stream is much more valuable than the one with occasional and sporadic customers despite the fact they may have the same value of assets and liabilities. So most sales, financial and management staff are familiar with the goodwill concept and I find it’s a good basis to sometimes assist in conveying what technical debt is and the affect it can have on their balance sheets. It’s also reasonably intuitive to a wider audience.

I like to think of technical debt as negative goodwill. Carry too much technical debt and the ability to deliver new features and customer satisfaction diminishes, which for a software company, reduces it’s viability and success potential. Technical debt is not an incidental, it’s a critical factor for the long term success of a software company. If your software cannot address customers needs and cannot be delivered in an acceptable time frame or cost then the business entity will soon cease to exist. This is why quantification of technical debt is so important, it may be vague, but until you’ve attempted to understand the costs of a decision, you can’t trade them off in a knowledgeable manner. I think a lot of poorly performing products are a result of people not fully understanding the technical debt they are accruing. Every piece of code has technical debt, this is expected and fine as long as the debt can be serviced by the organisation.

One pertinent example I like to use is that technical debt can spiral out of control just (well kindof) like the sub prime meltdown. There were lost of small loans (modules) which were very risky. Individually they were comparatively small amounts of debt so were manageable and were an acceptable debt. Agencies then bought up and packaged into larger units (made into an application) all of the smaller debts and mixed them with less risky debts which meant that the ability to see and quantify the real cost of the debt was lost. This tends to be true for applications, internally they may be sawdust, glue and paper, but on the outside it looks the same as every other application, so appears less risky that it really is. When conditions changed aversely in the market the true debt and risk became apparent and it all fell apart very quickly in a cascading manner. A good deal of the ramifications of the sub prime crisis extended beyond the US as outside bodies that had previously had good governance and risk management were unable to quantify and understand the real cost and risk of the debt. If you want to avoid a subprime meltdown in your own codebase it’s important to understand, quantify and actively manage technical debt.

So how do you measure technical debt? The first things is to acknowledge that it’s a nebulous concept and it’s really about estimation. You can live with an untested and poorly implemented piece of code or library but it’s going to be a source of errors for customers until you fix it. If you want to quantify the technical debt, start planning the project in your head to fix it in the next release. You’ll end up with costs like; the cost to write the new software in resources and time, the cost of ongoing support and fixes, the cost of developing patches and hotfixes, … . I like to use the thrown estimate model, pick a task, get a few wise heads, and each come up with an estimate in workdays then take the average.

Basic risk management techniques are also useful. Severity x Probability = Cost. Something terrible like a bug that causes 10% of your customers to lose data might be exceedingly rare, but the cost is monumental. Having code that could cause this because of a lack of investment in testing would be a high technical debt cost very few businesses with customers could afford. The more contingencies and planning you can do, the better you’ll be in a position to understand the real tradeoffs and convey that in terms that business understands. It’s also worth noting that you can go overboard, at some point you do need to ship, and not shipping also has a huge cost.

There’s much harder examples that don’t really lend themselves to quantification, for example; what is the technical debt of a product where getting user feedback is too slow, what is the cost of a hacked together and non intuitive user interface? My favourite at the moment, what is the debt of having distributed development teams in different timezones? The main thing is identifying what, if any, debt is being caused. It may not be able to be quantified specifically, but it’s often intuitive to people that something can have a intolerably high cost and must be addressed. That is, if the UI is just rushed out without proper feedback and testing cycle people won’t buy our software as it’s non intuitive and ugly and we’ll get poor reviews. If our install is terrible because we didn’t have the knowledge and didn’t sit down and plan it, then despite the fact our software is great, people won’t ever be able to install it, hence won’t know that.

I find thinking about choices in terms of debt and trying to quantify the costs often helps also cost the solutions. Eg What is likely to happen if there’s poor communication in distributed teams -> disjointed and poorly performing software with a high ongoing maintenance and rework cost -> what is the cost of outfitting a room in each location so people can do daily face to face meetings? and so on. In debt terms it’s just as important to cost the solution which is easier in some cases as it’s specific actions which usually means someone has to pay for it. There’s always the choice of doing nothing and you don’t want to start something if the future debt will be higher than the foreseeable benefits or that you can manage. What is the debt of the decision you are making when you write that line of code or choose that architecture?

This may sound like it’s a lot of overhead but not every decision needs to be quantified. Every dog has his day and every project has points where a decision which will define the success of the software has to be made. It tends to involve meetings with business owners, senior management and users. Going into the meeting with the options and quick dot points about the future costs and debt associated can really help to focus on getting the optimal decision.

Good Code, Bad Things February 2, 2008

I’ve taken over a legacy product in my current role and my job is with a new team to productise the good bits, drop the bad bits and make it so others can deliver applications and modules on top of the core with lower time and costs. It’s got quite a long history and was never really ‘productised’ and is about to become one of the core products of the company. It was developed by developers who were told to ‘just get it done’ which has meant that it’s devolved into a not so good piece of software.

This has got me thinking about the code I’ve worked on over the years which I think I can narrow down to three categorisations.

Bad code doing bad things, I think this comes from junior or naive programmers, the code is not understandable and it does things in what would be considered worst practice.

Good code doing bad things, the code is commented and understandable but it does something that gives you queasy feelings, eg sql injection, intermingling data access and presentation, no source control, no unit tests, quick fixes where no one’s really tried to understand why ….

Good code doing good things, the optimum, the code is well documented, efficient, semantically grokable and it solves the problem with the minimum of code and communicates and encourages good code to be built on top of it. Interestingly it’s usually less code in my experience.

I think over your career you write all three. It’s sometimes hard to admit, but at some point everyone has written bad code, it might not have reached production or you’ve gotten queasy half way through and taken a different approach, or it’s in production right now and it’s keeping you up knowing what a egregious hack it is. So how do you get a team writing good code to do bad things to be one where they write good code doing good things?

For starters start a review, mentoring and design process. I think often the bad things comes out of lack of time, lack of mentoring, lack of knowledge or sometimes lack of passion.  Someone on the team has to understand technical debt and convey this to the business.  People will in most cases try to do the right thing if given the opportunity. The ability to acquire and grow knowledge is a fundamental tenant of software engineering. For most passionate people in the industry they relish the challenge and if given the opportunity will take it with both hands.

Secondly start writing things down, start an operations manual for the dev team, start a wiki, communicate the how and why, build some organisational knowledge. If people coming into the team, especially graduates, can’t see the how then they’ll be doomed to repeat the process and never understand the why except from the hard lessons they’ll experience on their own. The scientific progress of human kind comes from standing on the shoulders of those who came before. If you’re not writing these things down and communicating them then you’re not part of the cycle of knowledge and people will be doomed to reinvention which is just a sad waste.

Play the devils advocate in reviews, don’t give people the answer but ask them how they would solve a problem with the implementation they’ve chosen, use a scenario to describe an extension to the problem or another view of the problem that their solution doesn’t cover. Ask how they could do this more efficiently, what about performance? This of course requires that you understand what it’s meant to do really well.

Always; positive, negative, positive. People respond better to encouragement than criticism, especially when creativity or passion is required to research and derive a solution. You don’t want new people on the team struck like bambi in the headlights.

These are some of the things I’ve found useful in teams I’ve been part of. I’ll be using some of these over the next 6-9 months to run the new team and reset the old habits in the delivery team so it will be interesting to see what works and what doesn’t.