Productivity metrics: garbage in, garbage out
Posted at 22:25 on 31 May 2008
I came across this article today when I was googling for a link for another blog entry. I was flabbergasted to see that it was written by someone with a PhD, appears in a professional engineering journal, and is currently linked from their home page:
Over time, there have been many attempts to define metrics that effectively measure software development productivity. Most of the ones that I have seen are amazingly complicated and very difficult to apply.
I think there is a simpler productivity metric which should be used across the industry: the total number lines of code in the organization divided by the number of people who are working on that code (including QA as well as development). For short, I will call this metric the LOC per head.
I propose that this measurement is an excellent representation of the development organization’s true productivity. If the number rises, it means that the development organization is more productive. If it decreases, it means that the organization is less productive
Ah, the old lines of code chestnut again. For some reason, managers seem to love it. The only problem is, it's totally brain-dead. Like government targets, any formal productivity metric can and will be gamed -- usually with disastrous results, as Joel Spolsky points out.
You want lines of code? Be prepared for your code base to be poisoned with endless copy and paste code and needless repetition, which, as any competent developer will tell you, is a nightmare to maintain. Or you may even end up with a joker on your team who decides to script the process and produce a million lines of code a second without even turning up at the office.
Besides, some frameworks such as Ruby on Rails or jQuery allow you to accomplish much more with far fewer lines of code. The first release of 37 Signals' Ta-Da List -- a full-blown commercial product -- contained less than 600 lines of Ruby code. So does that make DHH and colleagues unproductive? Of course not! On the contrary -- it makes them brilliant.
You want lots of check-ins to source control? Fine, you'll end up with dozens of them just to correct a single spelling mistake -- and as a side effect, a version history that leaves everyone totally confused as to exactly what's been going on.
You want lots of bug fixes in the issue tracker? Expect your developers to deliberately write bugs into their code so that they can "fix" them.
You want to compensate for this by penalising bug reports? You're asking your developers to mislead your testers about what functionality is actually in the code base so they'll pick up on fewer bugs.
And so on, and so on.
As the old computing adage goes, garbage in, garbage out.