When technical debt becomes technical bankruptcy

Posted at 07:00 on 06 April 2009

When clients and managers insist on a quick and dirty fix, they usually think that this means we only implement the "must-have" functionality, and leave aside the "nice-to-have" and even some of the "should-have" stuff. Okay, so we might need to reboot the server once a month or so, or we might need to go directly into the database using SQL Server Management Studio whenever prices change, but we can live with that, they think.

However, there is an unseen and potentially much more damaging effect of the quick and dirty fix, that developers, managers and clients all need to be aware of: technical debt.

Martin Fowler explains it like this:

You have a piece of functionality that you need to add to your system. You see two ways to do it, one is quick to do but is messy - you are sure that it will make further changes harder in the future.The other results in a cleaner design, but will take longer to put in place.

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

Now there are times when technical debt is an appropriate strategic decision. Often, you need to get to market quickly with your website in order to obtain a competitive advantage. Sometimes, there are hard deadlines to meet. However, just as with financial debt, you need to be aware that you are taking on a certain amount of technical debt, because it will need to be repaid sooner or later.

Consequently, when you make the strategic decision to accrue some technical debt, you should always allow for some time to repay it at regular intervals -- refactoring, adding more unit tests, documenting the system, and maybe even rewriting some components from scratch a second time if they are particularly messy. If you don't, and the situation is allowed to get out of hand, you can end up with technical bankruptcy.

Technical Bankruptcy is the point at which levels of technical debt become so overbearing that the project is effectively doomed. This is due to two serious and crippling effects of technical debt:

1. Making changes becomes much harder.

One of the quickest ways to build up technical debt is to copy and paste large sections of your code. Programming purists rightly regard copy and paste as evil -- it violates the "once and only once" rule and makes it much harder to make changes later on. However, it can add extra functionality into your program much more quickly than doing things the "correct" way. Many of the other programming anti-patterns, such as magic numbers, meaningless method names such as "do_it", and so on, fall into the same category -- shortcuts which involve a trade-off between getting things done now and getting things done in the future.

However, these are the less complex cases. More serious are the higher-level shortcuts: poorly designed database schemas, tight coupling between the tiers of the application, a lack of preliminary analysis and design, and so on. Nevertheless, whatever the cause, the result is that by asking for quick and dirty hacks today, you are potentially blocking yourself off from asking for quick and dirty hacks tomorrow.

2. Finding developers to work on the project becomes much harder.

Make no mistake about it: working on a project that has an excessive amount of technical debt is a depressing, discouraging, and disheartening experience. It can mean going for days or even weeks on end banging on the keyboard with little to show for it and getting shouted at by your boss for not delivering on time. It saps morale and by extension productivity, and this can spill over onto other projects and pull the rest of the team down. And what do developers do in that kind of situation? If they're any good, they go and find somewhere else to work.

This compounds the problem even more. When a developer leaves, you have to recruit someone else to fill their place. This means placing an ad wherever you normally place ads, spending time filtering CVs, conducting phone screens, technical tests (you do give potential hires a technical test of some nature, don't you?) and interviews, and then once you find a suitable candidate, there is the ramp-up time where they are actually getting to grips with the system -- and no prizes for guessing what technical debt does to the length of this ramp-up time.

Although we are in the throes of a recession, there is still a massive shortage of good developers. It's amazing just how many working programmers there are out there who can't even handle the most basic challenges such as the FizzBuzz problem, and who get by merely by copying and pasting SQL injection vulnerabilities from ten year old PHP tutorials, or by pestering their more talented and knowledgeable colleagues for help at every step of the way and proving to be a net drain on the company's resources. These are the kind of developers who have no concept of technical debt in the first place, and will just make things worse faster.

I would say that when you start to find it difficult to get good developers to work on a project for any length of time, it is a sign that your project is in serious trouble.

Is a total rewrite the answer?

Managers are invariably suspicious whenever a developer recommends a complete rewrite. Joel Spolsky wrote an article a while ago asserting that complete rewrites are a bad idea, and every time I say that X needs to be rewritten, this article is quoted back to me as a stock excuse for declining my suggestion.

Now the managers have a point here. A total rewrite from the ground up is the nuclear option: you run the risk of throwing away a whole lot of undocumented business rules, corner cases and bug fixes, and a complete redesign of the administrative interface means staff re-training and a deluge of support calls. Besides, developers can be a bit trigger-happy when calling for a total rewrite. We tend to be a pretty opinionated bunch, and very often we will call Someone Else's Code complete garbage simply because we don't understand it properly. But given a bit of time, most developers will manage to settle down into a new code base and will be able to start improving it and cleaning it up by a process of small, incremental refactorings as they go along.

Nevertheless, once you reach the point of technical bankruptcy (where technical debt becomes so overwhelming that the project is effectively doomed), a total rewrite might save the day, and maybe even provide some added value by clearing out certain business rules and systems that were put in place only as workarounds for bad design decisions in the first place. However, it would take a brilliant and determined developer, probably working on it as a skunk works project, to pull it off. Best to avoid getting into the situation in the first place, by paying down your technical debt sooner rather than later, and only hiring developers who know what they are doing to work on the project.