james mckay dot net
because there are few things that are less logical than business logic

April 2011

Eight wrong reasons why you are storing passwords for clear text recovery

The recent data loss by Sony of seventy million PlayStation users’ personal details was a serious lapse. They have assured us that their users’ credit card details were encrypted, though Graham Cluley of Sophos has asked questions about exactly what kind of encryption they were using.

A more serious issue is that they admitted that users’ passwords were compromised. This can mean only one thing: they were storing passwords in a form that could be recovered to plain text. The press doesn’t tend to make as much noise about passwords as about credit cards, which is a pity, because passwords are at least as valuable to a fraudster, if not more, for the simple reason that the overwhelming majority of people re-use passwords on multiple websites, including high-value ones such as their e-mail and their bank.

It’s a sobering reminder to all of us web developers that we should not be implementing plain text password recovery, period. It is inexcusable. And, it could be argued, illegal in some jurisdictions.

It is inexcusable because it is a security compromise that is totally unnecessary. Yet it is sometimes mandated by pointy-haired bosses and clueless clients despite developers’ objections. There was a question on Stack Overflow a while ago asking how you can ethically approach storing passwords for later plain text retrieval. I replied that you can’t.

Here are some of the excuses that people make for storing passwords in plain text.

1. “My site is small and obscure, nobody is going to try hacking it.”

If you are thinking that way, I have bad news for you. Hackers love small, obscure websites precisely because their developers think exactly that, and they are consequently riddled with SQL injection vulnerabilities exposing plain text passwords. Sites like that are low hanging fruit to a malicious user.

Furthermore, hackers operate botnets that trawl the Internet probing every website they come across for security holes. Even the smallest, most obscure website should expect to be probed by malicious scanners weekly if not daily.

How small and obscure is “smallest and most obscure”? How about my own blog, circa December 2005? At the time, it had been going for only seven months, and since I was (a) a lot more reserved and introverted back then than I am now, and (b) a complete blogging newbie, I had kept a very low profile with it up to then. Only half a dozen or so of my close friends knew of its existence. I had even turned off the feature to ping Technorati and other blog aggregators. Yet there it was, on the receiving end of an SMTP injection probe.

Bottom line: if you are collecting user registrations, you are not small and obscure. Period.

2. “Users’ logins to my site are a low-value asset.”

No they are not.

No matter how much we nag them, how much we tell them to be careful, people will still re-use passwords. There are far too many people who, when you tell them they need to use KeePass or something similar, will just shrug and say to you, “Oh, that’s too complicated for me, I’m not a computer person.” They regard good password practices as a tinfoil hat, because, after all, you’re not Fort Knox.

Consequently, if your password database gets breached, your attackers have not just got away with your users’ logins on your site. They have got away with your users’ logins on Facebook, eBay, PayPal, GMail, and their bank accounts. Give an identity thief access to your clients’ e-mail, and you have given them access to their entire life.

Far from being a low-value asset, your users’ logins are the most valuable asset that you are dealing with. They are arguably even more valuable than their credit card numbers.

3. “A password reset is harder from a usability perspective than a password reminder.”

No it is not. Not if you implement it correctly.

People only think that a password reset is harder to use because they think you have to e-mail your users a new password, and it usually ends up looking like transmission line noise. But there is a better alternative.

E-mail your users a one-time, time-limited link to a page on your website that lets them choose a new password then automatically logs them in once they’ve done so. This is actually easier to use than either a password reset or a password reminder, since one click from their e-mail client opens their web browser straight to the “choose a new password” page.

If you want to see this in action, this is how Amazon does it. So does Twitter. So does any other website that takes security seriously these days.

4. “As an administrator, I need to be able to log on as any other user, so that I can troubleshoot if need be.”

Regardless of whether this user story is a good idea or not (it might be controversial if, for instance, you are Facebook and word of it gets out), you do not need to know the other user’s password in order to facilitate it.

All you need is a button on your user administration page that allows an already authenticated superuser to switch identity to the user in question with a single click of the mouse. It’s exactly the same principle as the Unix su command.

Any web developer who isn’t able to implement this is, quite frankly, incompetent.

5. “Two-way encryption is sufficient.”

No it is not.

If you want to allow your users to recover their passwords through your web front end, your web front end has to be able to decrypt them. If your end users can decrypt their passwords, there is always the possibility that a malicious user can do likewise. If your code has a bug in it that lets a malicious user view passwords to which they are not entitled, that would totally nullify the effect of the encryption. Furthermore, you would have to put both your public key and your private key on a public-facing web server, making it all the easier for an attacker to get hold of the means to decrypt your users’ passwords.

6. “But what about credit cards? You need two-way encryption for them, surely?”

Your users’ credit cards should not be recoverable to plain text by the web server either.

If you look at how Amazon stores your credit card information, you will see that they don’t display it back to you once you’ve typed it in, apart from the last four digits. Yet they need access to this information in order to bill you correctly. So how do they do it?

The answer is simple. The web server only has access to their public key. They put the private key (and their back-end administration system) on a completely different server in a completely different subnet, separated from their web servers by firewalls with strict security.

If you want to store credit card details for later processing, this is what you must do. If you can’t do so, you should use a third-party payment provider such as PayPal, send their credit card details to them using web services, and do not store them in your own database.

Despite this, this is still not as secure as a one-way salted hash, and there is still a chance that an attacker could steal your users’ credit card database and the private key. But in this case there is a limit to how
much security you can practically apply without rendering the system unusable. This limit does not apply to passwords, since clear text recovery, as we have seen, is completely unnecessary for passwords.

7. “You need to do a risk assessment before deciding what security is appropriate.”

This one came up on a Stack Overflow discussion a while ago and I just about hit the roof when I saw it.

For crying out loud people, this isn’t a hypothetical, theoretical scenario! It is a scenario that has happened in practice, over and over and over again. It is a scenario that is keeping seventy million PlayStation users awake at night at this very moment. The risks are well known and well documented. If your users’ passwords get compromised, hackers can take control not only of their login to your dinky little website, but also to Facebook, PayPal, their e-mail, their bank. You are exposing them to the risk of identity theft, which can get pretty nasty.

Besides, a risk assessment is only meaningful if you can demonstrate that there is a convincing business case for any security compromises that you have to make. And what compromises do you have to make in order to avoid sending plain text passwords? There aren’t any! I’ve explained above just why the perceived disadvantages are not disadvantages of the sort, and how recoverable passwords are totally unnecessary for a user-friendly, flexible authentication system.

8. “The client/my boss insists on it.”

The best way to approach this is to show them how Twitter and Amazon handle login recovery (“show, don’t tell”), build a prototype, and point to some real-world examples of where things went wrong — the 4chan/Christian dating website that I mentioned a couple of years back is as good an example as any.

If, in spite of this, and being advised that they are probably breaking the law in requiring plain text passwords, they still insist on having passwords recoverable to plain text, you are likely to have other problems with your client. The Daily WTF kind of problems.

If you end up in this kind of situation, sack your client. Or start looking for another job if you’re a permie. Now.

So are there any valid reasons why you can be storing passwords for clear text recovery?

Yes, there is one. And only one. It is this.

You have inherited a legacy application, written by someone else, that has so much bad design and technical debt that it is infeasible to migrate your passwords to an encrypted version.

If you end up in such a situation, get the code under source control before you do anything else, to cover your back if nothing else. Advise the client that they are storing passwords dangerously. Warn them of the implications. Propose an alternative, and quote for it. And recommend that they make it a priority.

Agile versus methodologies

If the Agile Manifesto says this:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Then why do some people describe Agile as “an umbrella term covering a number of methodologies, such as Scrum and XP”? It sounds like a bit of a contradiction in terms to me.

Not that there’s anything bad about methodologies of course — it’s just that it seems to me that the whole point of the Agile Manifesto is to put them firmly in their place.

The purpose of auditing is to answer questions

Auditing is often seen as mindlessly bureaucratic and something of a waste of time imposed on us by regulators and compliance wonks. However, used effectively, auditing can be a powerful tool.

The purpose of an audit trail is to answer questions. Obviously, it is most effective if (a) it is set up to answer questions that people are actually likely to ask, (b) you are aware of those questions and bear them in mind when you’re actually filling in the form, and (c) your audit trail gives you tools to help you data mine the answers to those questions easily.

Source control is one example. Some people like to set up their source control policies so that every changeset has to be referenced to an item in the issue tracker, so that they can answer the question, “Which changesets fixed this bug?” While this is useful information to have, it isn’t the kind of question that people ask all that often. The kinds of questions people are likely to ask from your source control system are (a) “Which changesets introduced this bug?” and (b) “Who wrote this line of code, and why?”

These are the overriding principles behind how often you should check in code, and it’s also why I’m a great fan of hg bisect.

Change management is another example. Change management is sometimes scorned by some agilists as overly bureaucratic and pointless, but it only becomes that way if it’s implemented without any specific questions in mind. Here are some questions that an effective change management process can answer, for example:

  • What services are running on this server?
  • Who would know about the services that are running on this server?
  • I’ve run into a problem while making a change to this service, has it happened before, and if so how was it fixed?

Most of these questions concern specific resources, such as servers, software, build images, subnets, routers etc. A good change management system will let you tie these specific resources to specific changes, and will let you quickly search through them and get answers to these questions.

Why merges can (and should) be automated

Long time Mercurial users will no doubt appreciate the new merge and conflict resolution dialogs in TortoiseHg 2.0. When you have some conflicting files, rather than making you go through them one at a time with no idea how many more there are to handle, you are given a list of them with options to help you merge them.

TortoiseHg Conflict resolution dialog

However, there is one feature of this dialog that will no doubt raise an eyebrow or two. Whenever a file has been modified on both sides of the merge, it reports it as a conflict, even if the modifications were to completely different parts of the file. What is going on here? Has Mercurial suddenly forgotten how to merge? Is it turning into Team Foundation Server? Whatever next, read-only files and baseless merges?

Actually, no it hasn’t. That was my reaction when I first tried out the development version of TortoiseHg 2.0 last summer, so I rolled up my sleeves and coded up an option to restore the traditional behaviour:

TortoiseHg options - Autoresolve merges

When you merge, you can also choose on a case by case basis between automatic and manual file resolution:

TortoiseHg merge wizard - autoresolve merges

So why does it work this way now? In my discussions on the TortoiseHg mailing list, Steve Borho, the lead developer of TortoiseHg, pointed out that there’s a lot of hallway usability testing behind it:

I’ll allow that long-time Mercurial users may find this limiting, so I’ve assumed that we’ll eventually add a back door to revert to default Mercurial behavior.  But I have heard from many new users over the years that this is the one part of the Mercurial interface that is unsettling, having kdiff3 thrown at them at seemingly random occasions, so I want the internal:fail approach to be the initial default.

André Sintzoff concurred:

I agree with you. Most of the new users I know are somehow disturbed by the “old” merge behaviour.

When I show them the “new” behaviour, they are enthusiast.

They had a valid point. This is something I’d forgotten about myself.

Inexperienced developers are usually terrified of merging. When you’re combining two people’s changes together, you need to know and understand not only the changes themselves, but the context as well. To delegate the entire process to some unknown computer algorithms sounds reckless and dangerous. This was one of the first things I found intimidating about svn update when I first started using source control in a team context in the first place.

Yet in practice, fully automated merging works remarkably well. When you run svn update or hg merge, more often than not, it all goes very smoothly — in fact, much more so than attempting to merge everything manually. Why should this be?

1. In the overwhelming majority of cases, the default option is the correct one.

Next time you do a merge, turn off automatic conflict resolution and use a three-way tool such as Perforce Merge. I particularly like Perforce Merge because it shows you exactly what’s going on. At the top, you have the two sides of the merge on either side of the original version, so you can tell whether something was added on the left hand side or whether it was deleted on the right hand side:

Merge resolution example

In the most basic case, automated merge tools assume that if a change was made to one side of the merge, but there is no corresponding change on the other side, that change should be included in the final result. That’s what shows up in the bottom pane. On the other hand, if two people have edited the same part of the file, it shows up as a merge conflict and you have to resolve it manually.

Now here’s the key. Once you’ve carried out a few manual merges, you soon realise that with non-conflicting text differences, you almost never choose anything other than this default option. It becomes evident that working your way manually through a string of differences where you only ever choose the default is largely a waste of time.

2. Manual merges increase the risk of human error.

Having said that, automated merges don’t always get it right, and you do sometimes need to be aware of the context on each side. But — and it is a big but — manual merges fare no better.

Here’s a simple example where both automatic and manual merges are liable to give the wrong result. Let’s say that two developers, Alice and Bob, both make an identical change to a source file on their respective branches — for example, throwing an exception when something can’t be found. Then, Bob commits a subsequent change which backs it out. Should the new code be included in the merge or not?

     Changed ------
    /              \
Original         ???????
    \              /
     Changed - Original

Mercurial and Git both take the line that because Bob undid the change, it should be as if he had never made it in the first place — a feature called “implicit undo” — and therefore, the change should be included in the merge. But that is not necessarily what you want. An ideal version control tool would report this as a conflict, but what happens then?

Here’s what it might look like in your merge tool:

Merge resolution with implicit undo

There is no indication whatsoever that as well as being added on the left hand side, that exception was also added on the right hand side and then deleted again. Because your manual merge is a naive three-way merge, with no awareness of history, it also gives you implicit undo, and unless you are particularly on the ball and aware that this change was made then undone in the first place, you won’t pick up on it.

But if you’ve just worked your way through a dozen or more diffs where you’ve chosen the default option every time, the chances are that your eyes will be glazing over, you won’t be on the ball, and you’ll miss it. And therein lies the rub: as well as being slow, manual merge resolution increases the risk of human error.

Another problem with manual merge resolution is that it frequently presents you with diffs that are pretty confusing and overwhelming. Visual Studio .sln files are a particular pain to work with in this respect, since you are dealing with lines and lines of GUIDs that blur into each other. Very often, the only difference between the two sides is that stuff has been moved around. In cases such as these, it can be almost impossible to carry out a manual merge effectively, whereas an automated merge will work out fine. Long lines just compound the problem.

So there’s your trade-off. An automated merge, which may or may not be correct due to ignorance of context. Or a manual merge, which may or may not be correct due to human error and lack of clarity of both context and content. And is several orders of magnitude slower into the bargain.

3. Semantic resolution is easier dealt with by compiling and testing anyway.

The upshot of this is that merging is actually a two-pass process, regardless of how you do it. The mechanical operation of combining your changes is not the be-all and the end-all, but only the first step. Once you’re done with it, you will need to test your merge and fix up any problems. But this isn’t a big deal — it’s the kind of thing you’re doing all the time in normal coding anyway.

Besides, manual resolution only gives you a narrow view of what you’re doing. It’s only when you compile and test that you really see how the two sides of the merge fit together and get a feel for how to deal with the context and intent of the two sides of the merge.

Most problems with merges show up when you attempt to compile your code. In these cases, it’s merely a case of fixing them up — clearing up ambiguous references, checking renames and so on. If you have good test coverage (and you should have good test coverage), your unit tests will pick up the majority of other problems, though you do need to be aware that incorrect merges may have an impact on your tests too. And while some problems may slip through the net, they generally are pretty insignificant in number and scope compared to bugs that creep in through normal, everyday coding.

Fully manual merge resolution is helpful for new users because it eases them gradually into the apparently scary world of branching and merging. But once you are used to it, it becomes apparent that there is little or no benefit to the all-manual approach. While you may feel more in charge of the process while you’re carrying it out, this is largely illusory and a waste of time, somewhat akin to premature optimisation. Provided that your tooling has decent automatic merge support — and Mercurial certainly does have decent automatic merge support — there’s every reason to make the most of it.