Why Stack Overflow’s reputation system is broken

Posted at 11:30 on 27 September 2008

I find it rather ironic that the author of the blog entry from which this excerpt is taken:

It seems like any time you try to measure the performance of knowledge workers, things rapidly disintegrate, and you get what Robert D. Austin calls measurement dysfunction. His book Measuring and Managing Performance in Organizations is an excellent and thorough survey of the subject. Managers like to implement measurement systems, and they like to tie compensation to performance based on these measurement systems. But in the absence of 100% supervision, workers have an incentive to “work to the measurement,” concerning themselves solely with the measurement and not with the actual value or quality of their work.

is also one of the faces behind a programmer website which does exactly what he is railing against.

I’m talking about the Stack Overflow reputation and badge system. Granted, it was more Jeff Atwood’s idea than Joel’s — he took his inspiration for it from the Xbox 360 — but the big problem is that when you try to turn a serious system that is supposed to be all about Getting Things Done into a game, people just game the system and turn it into an unusable mess that is not fit for purpose.

If you want to see what I mean, just take a look at this question, which I asked yesterday afternoon. I’ve been looking for a bug tracker system which can work as an integrated system for both developers and project managers for a while now, and none of the ones I’ve looked at so far have the particular feature I’m asking for.

The first so-called answer came within seconds and didn’t answer the question properly, which isn’t surprising since you would need at least 2-3 minutes just to read the question in the first place. It was followed by a string of about ten or so responses over the next half hour, again, very few of which made much effort to read the question, let alone answer it. Most people seemed to treat it as saying “What is your favourite issue tracker?” and one busybody even tagged it as “subjective” when I was asking for something very specific. And nobody so far has reported any success or otherwise with using an issue tracker of any description to integrate both the developer’s-eye view and the project manager’s-eye view.

This is a BIG problem with Stack Overflow, and I’ve seen it to an extent on other questions too. The system doesn’t favour good answers or correct answers or answers that actually make any attempt to answer the question, it favours quick answers. Being the first off the mark with something that at least looks like it could plausibly be an answer to the question means you’re most likely to get voted up. Getting voted up means appearing at the top of the list of answers, and it’s kind of self perpetuating because then you get more votes, and each vote means that you get ten reputation points, and if you get enough reputation points, you automatically become the Stack Overflow equivalent of a Wikipedia administrator.

The result is that you get a whole lot of knuckleheads gaming the system trying to pimp their reputation. They put up a response that looks fairly plausible and seems right to other knuckleheads but which either (a) doesn’t answer the question, or (b) is plain wrong. If the person asking the question is also a knucklehead, their answer gets marked as the accepted answer, which means even more reputation points. In the meantime, someone who arrives several days or weeks later with the correct answer doesn’t get any attention because their answer gets buried in all the other zeros. It’s particularly worrying because it’ll be the knuckleheads who end up running the show and deciding what goes and what doesn’t.

It’s as broken as lines of code per day, and it really really annoys me.

It really annoys a lot of other Stackers too — a request to fix it is the most popular user request on the Stack Overflow uservoice forums, though the problem is that there is no consensus about what needs to be done to stop it. I do hope they come up with some fix for it, otherwise the site could end up with no more value than its arch-nemesis, expertsexchange.