james mckay dot net

There are two types of programmer in the world.

Now when somebody writes that on their blog, they’re generally criticised as being elitist and condescending. This is because they tend to talk about the “alpha geeks” or the “20%,” and everyone else or the “80%.”

I’m not going to say that. But I am still going to say that there are two types of programmer.

• Those for whom programming is a means to an end.
• Those for whom programming is an end in itself.

If you’ve been following me on Twitter, and if you’ve read my recent blog posts, you’ll know that this has been something of a theme for me for the past couple of months or so. What got me going about it was reading this remark in a blog post by Jeremy Miller a few months back:

I had a great time at CodeMash yet again. For those of us who look at software development as more of a lifestyle than just a way to meet the mortgage, these kinds of events are like a huge dose of nutrition for the soul. I really liked Scott Chacon’s keynote address and it’s got my head a going about how to make our workplace better (see this again).

It’s not a bad post, all in all, but whoa there. Since when was software development supposed to be a lifestyle?

Don’t get me wrong here. I’m all for being passionate about programming, for sharpening the saw, for improving your skillset, for using the best tools for the job, and all that, but when I hear of people describing software development as a lifestyle, I wonder just what makes them tick. And this isn’t a criticism of Jeremy either. It’s more a criticism of myself. There have been times when I myself have slipped into that way of thinking, and I’ve not found it to be beneficial. It’s easy to fall into that trap when you have the kind of mind that enjoys programming, and it takes a good deal of self-discipline to avoid it. In fact, paradoxically, I find that when I get into programming-as-an-end-in-itself mode, I end up getting less code written, not more.

This, in a nutshell, is what’s been behind a lot of the things I’ve been blogging about recently. It’s been behind my recent decision to stop using vim, for instance. For that, and everything else I’ve been looking into, I’ve been asking myself the question: is this a means to an end, or is it an end in itself?

So to those of you who would want to persuade me to reconsider learning vim properly, or to switch from Mercurial to git, please allow me to remind you of one important fact.

You can’t learn everything.

As programmers, we face a firehose of new tools, new technologies, and so on. Many of these are promoted loudly with great fanfare on the Internet by their fans. In the case of some (*cough* git *cough* ruby on rails *cough*) the fanboy hype is deafening. Most of them are claimed by their proponents to be counterexamples to Fred Brooks’ classic essay, No Silver Bullet. Almost none of them are anything of the sort. Yes, they are beneficial to a greater or lesser extent, but in most cases, there are trade-offs.

It simply isn’t possible to give all of them more than a cursory once-over. You have to triage what to learn and what not to learn. Aggressively. Learning vim would rob me of time familiarising myself with other, more important technologies such as Selenium or backbone.js, for instance. Frameworks and tools that you do learn are huge and complex these days, and if you try to learn too many of them, you end up with a very superficial understanding of them that can lead to problems. Either that, or else you end up spending so much time on the computer that you damage your health and your relationships with other people.

So in the past few weeks, I’ve been taking a good hard look at everything I’m doing, re-evaluating what I should be taking up, what I should be dropping, and what I should just allow to carry on as-is. With that in mind, I’ve been formulating some criteria by which I should decide what to adopt, and what to ignore. With everything I’ve been doing, I’m asking myself a few pertinent questions now:

• Does this solve a problem that I am currently facing or am likely to face in the not too distant future? If so, how effectively?
• Will knowing it enhance my career prospects? More importantly, will not knowing it harm my career prospects?
• Are there any examples of real-world applications using it, or is it all smoke and mirrors?
• What is the ramp-up time, and is it worth it?
• Is it in line with the direction I am heading, or is it a distraction?
• If it makes programming easier, does it do so at the expense of the end users?

The last point is a key implication of programming being a means to an end rather than an end in itself. If programming is a means to an end, and you care about your end users, you will base your choice of tools and frameworks on what serves them best. On the other hand, if programming is an end in itself, you will choose the cool languages and tools that everyone is raving about, but at the expense of your audience. This is of course what WordPress got right but Diaspora got wrong.

I’ve decided that in terms of tools and technologies, I should focus on my core competencies: namely, the Microsoft .NET stack, HTML 5 and JavaScript. I also want to up my game as far as test-driven development is concerned — hence, I’ll probably have another, closer look at MSpec — and also get to grips much more with agile development methodologies, in particular Scrum, which we have been using at work for the past year or so. As far as my other skills are concerned, such as PHP, Python, Django and Linux, these are useful skills to have, but how much time I spend on them from now on is something I am yet to decide. While it’s good to diversify to an extent, I don’t think much of the Pragmatic Programmers’ suggestion that you should learn another language every year, simply because that takes you into the realm of programming for programming’s sake, and besides, I don’t think it’s realistic, unless you are content to be a jack of all trades and master of none.

Bloggingggaaaaaaaa!

The ALT.NET movement makes me feel a little bit uneasy somehow. While there are some pretty good people in there, and I fully support the aim of finding better ways to do things, the movement as a whole seems to have largely degenerated into a clique of programming-as-an-end-in-itself Ruby fanboys with inflated egos who only do .NET under protest for their day jobs, and who complain about Microsoft tools purely for the sake of complaining about Microsoft tools. I ended up in conversation on Twitter with several of them in the wake of my TFS==Lotus Notes blog post, and I think some of that may have rubbed off onto me. Certainly, I’ve been doing far too much complaining recently, and I’m sure it doesn’t make me look good.

I’ve come to the conclusion that the programmers most worth following on Twitter, most worth subscribing to in Google Reader, are the people who are building end-user applications aimed at non-developers. This isn’t necessarily because they’re smarter — frameworks and developer tools are technically more challenging beasts to write and maintain — but because they tend to have a better grip on reality. Building end-user applications puts activities such as socialising and speaking to non-programmers right at the heart of programming: you have to talk to people before you start in order to get ideas, and then you have to talk to them afterwards in order to make sure you’ve delivered something that they can use without getting confused. And of course, you also have to talk to them in between times to make sure you are on the right lines. This is where agile software development hits the nail on the head — developers are expected to talk to the customers on a daily basis. On the other hand, only ever building programming tools and libraries, and little or nothing else, seems a bit self-serving to me.

At the same time, I’ve also been asking the same questions about my blog. Over the past few years, I’ve constantly found it to be a very time consuming animal, where I don’t see much benefit coming out of it, and I keep wondering why I do it. Most of my recent blog posts have said, “You should be doing this,” or “You shouldn’t be doing that,” or “This is a cool tool,” or “That process is bad,” or other things in that vein, and I’ve been saying very little about what I’ve actually been doing myself. This is just punditry, and I don’t want to be just another talking head. We have too many talking heads in our industry, and it gets rather boring after a while.

I also think I got carried away somewhat with the whole Mercurial/source control thing. Now don’t get me wrong: I still maintain that Mercurial is the best source control tool available today by a long shot, that only git comes anywhere near it, that the old-school centralised model is past its sell-by date and no longer fit for purpose, and that if you’re not using a DVCS, you’re harming your productivity, making a rod for your back, and exposing your project to unnecessary and unacceptable risks. But there comes a time when you have to let things drop, otherwise you end up coming across like a noisy fanboy yourself and you just annoy people who have missed them as they flew straight past coming out of the firehose. In fact, this fanboy attitude is totally out of order in the Mercurial culture itself, where viewing programming (and by extension, source control) as a means to an end and not an end in itself is very much the norm.

I’ve therefore decided that I need to take a break from blogging for the next few months. Quite when, or even whether, I return is anyone’s guess, but if I do, I’m not planning to carry on writing in the same vein as I’ve been doing so up to now. If I’m to carry on blogging, I want to build stuff that I can talk about and showcase here on my blog first. Unfortunately, at the moment, most of the code I write that’s worth talking about is at work, and there’s obviously a limit to how much I can write about that. But the Agile Manifesto tells us to favour working software over comprehensive documentation, and as a corollary, the contents of your github or bitbucket account say more about you as a developer than the contents of your blog.

I’ll no doubt continue working on some personal programming projects in my spare time. In fact, hopefully I’ll be able to spend more time working on programming projects without getting distracted by blogging about x, y or z. But it will have to be time-boxed anyway. During the week, I am out of the house from seven in the morning until seven in the evening, which doesn’t leave me much time for anything other than sleep, and I want to free up more time for socialising, church activities, and developing other interests that don’t involve sitting over a computer.

Comments (2) »

Posted at 08:00 on 3 October 2011.
Tagged: blogging, programming

Here are a few of my random thoughts on Ayende’s oh so controversial post, If you don’t have pet projects, I don’t think I want you:

1. Pet projects don’t need to be extensive or advanced for you to stand out from the crowd. The vast majority of .NET developers have no personal projects whatsoever, so as little as one evening a month spent on them should be more than adequate.

2. Can we cut out all the cringeworthy Bravo Sierra about “passionate developers,” please? There are less clichéd ways of saying much the same thing, and besides, I get the impression that some recruiters view passion as a substitute for competence. It isn’t. You get plenty of passionate developers who think that Pokémon exception handling is perfectly acceptable, for instance.

3. It’s also possible to have too much passion for programming. People who get too passionate about programming forget that it is a means to an end, and not an end in itself. A telltale sign that you are dealing with such a person is that their github account contains nothing but vim scripts and clones of developer tools and class libraries. If, as a recruiter, you want to look for personal projects, look for ones aimed at non-developers. That way, you get some indication that you’re dealing with someone who at least has some grip on reality.

Comments (1) »

Posted at 08:00 on 15 September 2011.
Tagged: programming

Github, so we are told, is the most popular source code hosting website on the planet. With nearly a million registered users and 2.5 million repositories, it is certainly impressive. But if you look at what kind of projects are actually hosted on Github, a certain picture emerges.

Take a look at its list of about eighty or so “interesting” repositories, for starters. There’s a lot of cool stuff there: Prototype, Scriptaculous, Ruby on Rails, MooTools, Groovy, Sinatra, Redis, Symfony, and so on. The list changes from time to time, and I have no idea what criteria are used to construct it, but at the time of writing, with only one exception  — phpBB — every single repository on the list is either a programming language, or a programming framework or library, or a programming tool.

You can see something similar — and quite surprising — when you look at the list of most popular languages on Github. The dominance of JavaScript, Ruby and Python is not all that surprising, but the fourth most popular language on Github is Shell. In other words, bash scripting — the Linux equivalent of DOS batch files or PowerShell.

C# — probably the most sought-after programming language by employers in the Real World — does not appear in the top ten. Clicking through to it from the right hand column shows that it is number twelve. But which is number eleven? After a minute or two of clicking on educated guesses then completely at random, I discovered, much to my surprise, that it is actually VimL.

I kid you not. Github apparently has more repositories dedicated to the scripting language for Vim — a console-based text editor mainly used by hard-core geeks — than to C#.

Why am I highlighting these things? Because they are indicative of a problem that seems to be plaguing the open source world and the programming blogosphere these days. There is far too much of a focus on processes and tools, and not enough of a focus on individuals and interactions. I’ve been getting too much inclined that way myself in the past couple of years here on my blog and on Twitter, and I’m getting weary of it.

You can understand why this would be. Programming languages, tools and frameworks are fun to work on because they’re hard and make you a better programmer, they scratch a personal itch, and they look good if you’re trying to attract the attention of trendy startups in Silicon Valley. And of course these things are important. But they are written by open source developers, for open source developers. They are of no direct interest whatsoever to non-developers.

Where are all the WordPress plugins and themes? The personal productivity software? Personal finance? Health and fitness? Photo and video editors? Education? Motoring? Foreign language learning? E-commerce? Geolocation? Astronomy? Twitter and Facebook clients? Games? Screensavers? Android and iPhone apps? Yes, they’re out there, but you have to hunt for them. And among all the blog posts about how CQRS, NoSQL and IOC containers are so cool, where are all the success stories telling us how they’re being used in the Real World?

Programming does not exist in a vacuum, folks. If we were serious about the Agile slogan, “Individuals and interactions over processes and tools,” projects aimed at non-developers would predominate. As it is, a lot of it seems like a case of programming for programming’s sake.

A lot of developers are introverts, and find it easier to spend time writing code than interacting with people. But if you want to produce something that’s really useful, you need to spend some time getting out from behind the computer, developing other hobbies and interests, and interacting with people. After all, that’s where the ideas for useful software come from in the first place.

Being a successful developer really does require you to put individuals and interactions over processes and tools.

Comments (4) »

Posted at 08:00 on 15 August 2011.
Tagged: programming

Here’s something I’d like to see. The ability to purchase train tickets — especially season tickets — at a supermarket along with my weekly shopping.

Queuing at a station counter or a ticket machine is something I only normally have to do once a month or so but all the same, it’s still a complete faff at quarter past seven in the morning, especially if, like me, you’re not a morning person. Getting to the station that extra bit earlier and ending up in a queue of twenty other commuters at that time of the morning when your train is due in five minutes is just a little bit stressful.

On the other hand, I pay a regular visit to Tesco or Sainsbury’s at least once a week. When I do, I have much more leeway with my time. It’s later in the day so I’m not bleary eyed and newly out of bed. Having to stand in a queue for ten or fifteen minutes doesn’t faze me nearly so much. It would be a much more seamless fit into your average commuter’s weekly routine.

Comments (2) »

Posted at 08:00 on 8 August 2011.
Tagged: commuting

It seems that Microsoft has announced that HTML 5 and JavaScript, rather than Silverlight, will be the basis for front-end applications in Windows 8. The Silverlight community is up in arms.

This decision is hardly surprising. Silverlight was pretty much doomed from the start. It was a rival to a well-established, if flaky, technology — Flash — and pretty late to the party, so to developers not wedded to the Microsoft ecosystem (and that means 90% of web developers and designers), it was a great big yawn. And its prospects were effectively killed off when Apple decided to ban non-approved programming languages and frameworks from the App Store — therefore, no Flash, and by extension, no Silverlight. The iPhone and iPad are a huge market — if something doesn’t run on them, it isn’t cross-platform.

In the meantime, HTML 5 has come to the fore as a standard that looks set to render both these technologies obsolete. It is (partially at least, and increasingly) supported natively by web browsers without requiring any additional extensions. It’s an open W3C standard with a history spanning two decades, so it’s here to stay, as well as being a skill that can easily be transferred to other environments. XAML may be nice, but it has little or no traction outside of .NET.

For what it’s worth, this illustrates the risk of limiting your experience and skills to the Microsoft ecosystem. A lot of people have invested a lot of time and effort in becoming Silverlight specialists, and now they’re scared because it looks like those skills are set to become pretty much worthless over the next few years. I wouldn’t advocate ditching Microsoft altogether, but it’s always a good idea to be attentive to what’s going on elsewhere and not put all your eggs in one basket. Besides, if you’re familiar with what’s going on elsewhere, it can help to give you a better feel for which of Microsoft’s frameworks and technologies are likely to be a good long-term investment and which aren’t.

Comments (34) »

Posted at 10:00 on 3 June 2011.
Tagged: HTML 5, Silverlight

Another wrong argument that occasionally comes up in favour of plain text passwords is that you shouldn’t be responsible for the fact that your users do stupid things, like re-using passwords. Unfortunately, it’s not that simple.

In the eighteen months since I started using KeePass to manage my online passwords, I’ve found that it involves a certain amount of friction. For starters, it’s less user-friendly. Rather than just typing your user name and password straight into your browser, you have to switch to the KeePass window, find your website, and then paste. There are shortcut keys and a search facility to make things easier, but it is a bit of a learning curve. Furthermore, when you register with a new site, you have to fiddle about with the password generator in order to create a new password, and on top of that, some websites have undocumented limitations or bugs in their password forms. One well known website that I use allows you to set passwords of any length, but limits you to 20 characters in the login screen, for example.

That’s a relatively minor complaint, of course. A much more serious difficulty is synchronising passwords between different devices, not all of which may support your password manager of choice. KeePass for the iPhone is not yet available outside the USA and Canada, for example. Third party password managers may not even be an option on certain other Internet-enabled devices such as the PlayStation, Internet TV, and so on. Then there are situations such as a friend’s computer when you don’t have your password database on you; Internet cafes and kiosks; and locked-down workstations on corporate networks.

Clearly, avoiding password reuse requires a certain amount of discipline, sacrifice, and technical know-how. Even many relatively tech-savvy users view it as one of those things that “I must get round to someday” — a bit like taking up exercise or flossing your teeth. So where does that leave your non-technical users?

The overwhelming majority of non-technical users are shockingly ignorant about even the most basic aspects of web use. A couple of years ago, Google interviewed passers-by in Times Square, New York, and found that more than ninety percent of the people they stopped didn’t even know the difference between a web browser and a search engine:

The upshot of all this is that as a data controller, it is totally unrealistic to expect your users not to re-use passwords. They shouldn’t re-use passwords, and they should use a password manager, and they should choose secure passwords, and you should warn them of the risks. But to many of your users, if you try to explain the risks to them and what to do about it, their eyes will just glaze over and they will say to you, “Oh, I’m not a computer person. That’s all too technical to me.” They will re-use passwords. In so doing, many of them will entrust you with the login details to their e-mail, Facebook, PayPal, and possibly even their bank accounts. They shouldn’t, but they do. And with that in mind, you have a responsibility to do everything in your power to protect those details.

Comments (3) »

Posted at 08:00 on 9 May 2011.
Tagged: passwords, security

The recent data loss by Sony of seventy million PlayStation users’ personal details was a serious lapse. They have assured us that their users’ credit card details were encrypted, though Graham Cluley of Sophos has asked questions about exactly what kind of encryption they were using.

A more serious issue is that they admitted that users’ passwords were compromised. This can mean only one thing: they were storing passwords in a form that could be recovered to plain text. The press doesn’t tend to make as much noise about passwords as about credit cards, which is a pity, because passwords are at least as valuable to a fraudster, if not more, for the simple reason that the overwhelming majority of people re-use passwords on multiple websites, including high-value ones such as their e-mail and their bank.

It’s a sobering reminder to all of us web developers that we should not be implementing plain text password recovery, period. It is inexcusable. And, it could be argued, illegal in some jurisdictions.

It is inexcusable because it is a security compromise that is totally unnecessary. Yet it is sometimes mandated by pointy-haired bosses and clueless clients despite developers’ objections. There was a question on Stack Overflow a while ago asking how you can ethically approach storing passwords for later plain text retrieval. I replied that you can’t.

Here are some of the excuses that people make for storing passwords in plain text.

1. “My site is small and obscure, nobody is going to try hacking it.”

If you are thinking that way, I have bad news for you. Hackers love small, obscure websites precisely because their developers think exactly that, and they are consequently riddled with SQL injection vulnerabilities exposing plain text passwords. Sites like that are low hanging fruit to a malicious user.

Furthermore, hackers operate botnets that trawl the Internet probing every website they come across for security holes. Even the smallest, most obscure website should expect to be probed by malicious scanners weekly if not daily.

How small and obscure is “smallest and most obscure”? How about my own blog, circa December 2005? At the time, it had been going for only seven months, and since I was (a) a lot more reserved and introverted back then than I am now, and (b) a complete blogging newbie, I had kept a very low profile with it up to then. Only half a dozen or so of my close friends knew of its existence. I had even turned off the feature to ping Technorati and other blog aggregators. Yet there it was, on the receiving end of an SMTP injection probe.

Bottom line: if you are collecting user registrations, you are not small and obscure. Period.

2. “Users’ logins to my site are a low-value asset.”

No they are not.

No matter how much we nag them, how much we tell them to be careful, people will still re-use passwords. There are far too many people who, when you tell them they need to use KeePass or something similar, will just shrug and say to you, “Oh, that’s too complicated for me, I’m not a computer person.” They regard good password practices as a tinfoil hat, because, after all, you’re not Fort Knox.

Consequently, if your password database gets breached, your attackers have not just got away with your users’ logins on your site. They have got away with your users’ logins on Facebook, eBay, PayPal, GMail, and their bank accounts. Give an identity thief access to your clients’ e-mail, and you have given them access to their entire life.

Far from being a low-value asset, your users’ logins are the most valuable asset that you are dealing with. They are arguably even more valuable than their credit card numbers.

3. “A password reset is harder from a usability perspective than a password reminder.”

No it is not. Not if you implement it correctly.

People only think that a password reset is harder to use because they think you have to e-mail your users a new password, and it usually ends up looking like transmission line noise. But there is a better alternative.

E-mail your users a one-time, time-limited link to a page on your website that lets them choose a new password then automatically logs them in once they’ve done so. This is actually easier to use than either a password reset or a password reminder, since one click from their e-mail client opens their web browser straight to the “choose a new password” page.

If you want to see this in action, this is how Amazon does it. So does Twitter. So does any other website that takes security seriously these days.

4. “As an administrator, I need to be able to log on as any other user, so that I can troubleshoot if need be.”

Regardless of whether this user story is a good idea or not (it might be controversial if, for instance, you are Facebook and word of it gets out), you do not need to know the other user’s password in order to facilitate it.

All you need is a button on your user administration page that allows an already authenticated superuser to switch identity to the user in question with a single click of the mouse. It’s exactly the same principle as the Unix su command.

Any web developer who isn’t able to implement this is, quite frankly, incompetent.

5. “Two-way encryption is sufficient.”

No it is not.

If you want to allow your users to recover their passwords through your web front end, your web front end has to be able to decrypt them. If your end users can decrypt their passwords, there is always the possibility that a malicious user can do likewise. If your code has a bug in it that lets a malicious user view passwords to which they are not entitled, that would totally nullify the effect of the encryption. Furthermore, you would have to put both your public key and your private key on a public-facing web server, making it all the easier for an attacker to get hold of the means to decrypt your users’ passwords.

6. “But what about credit cards? You need two-way encryption for them, surely?”

Your users’ credit cards should not be recoverable to plain text by the web server either.

If you look at how Amazon stores your credit card information, you will see that they don’t display it back to you once you’ve typed it in, apart from the last four digits. Yet they need access to this information in order to bill you correctly. So how do they do it?

The answer is simple. The web server only has access to their public key. They put the private key (and their back-end administration system) on a completely different server in a completely different subnet, separated from their web servers by firewalls with strict security.

If you want to store credit card details for later processing, this is what you must do. If you can’t do so, you should use a third-party payment provider such as PayPal, send their credit card details to them using web services, and do not store them in your own database.

Despite this, this is still not as secure as a one-way salted hash, and there is still a chance that an attacker could steal your users’ credit card database and the private key. But in this case there is a limit to how much security you can practically apply without rendering the system unusable. This limit does not apply to passwords, since clear text recovery, as we have seen, is completely unnecessary for passwords.

7. “You need to do a risk assessment before deciding what security is appropriate.”

This one came up on a Stack Overflow discussion a while ago and I just about hit the roof when I saw it.

For crying out loud people, this isn’t a hypothetical, theoretical scenario! It is a scenario that has happened in practice, over and over and over again. It is a scenario that is keeping seventy million PlayStation users awake at night at this very moment. The risks are well known and well documented. If your users’ passwords get compromised, hackers can take control not only of their login to your dinky little website, but also to Facebook, PayPal, their e-mail, their bank. You are exposing them to the risk of identity theft, which can get pretty nasty.

Besides, a risk assessment is only meaningful if you can demonstrate that there is a convincing business case for any security compromises that you have to make. And what compromises do you have to make in order to avoid sending plain text passwords? There aren’t any! I’ve explained above just why the perceived disadvantages are not disadvantages of the sort, and how recoverable passwords are totally unnecessary for a user-friendly, flexible authentication system.

8. “The client/my boss insists on it.”

The best way to approach this is to show them how Twitter and Amazon handle login recovery (“show, don’t tell”), build a prototype, and point to some real-world examples of where things went wrong — the 4chan/Christian dating website that I mentioned a couple of years back is as good an example as any.

If, in spite of this, and being advised that they are probably breaking the law in requiring plain text passwords, they still insist on having passwords recoverable to plain text, you are likely to have other problems with your client. The Daily WTF kind of problems.

If you end up in this kind of situation, sack your client. Or start looking for another job if you’re a permie. Now.

So are there any valid reasons why you can be storing passwords for clear text recovery?

Yes, there is one. And only one. It is this.

You have inherited a legacy application, written by someone else, that has so much bad design and technical debt that it is infeasible to migrate your passwords to an encrypted version.

If you end up in such a situation, get the code under source control before you do anything else, to cover your back if nothing else. Advise the client that they are storing passwords dangerously. Warn them of the implications. Propose an alternative, and quote for it. And recommend that they make it a priority.

Comments (2) »

Posted at 09:00 on 29 April 2011.
Tagged: passwords, security

If the Agile Manifesto says this:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Then why do some people describe Agile as “an umbrella term covering a number of methodologies, such as Scrum and XP”? It sounds like a bit of a contradiction in terms to me.

Not that there’s anything bad about methodologies of course — it’s just that it seems to me that the whole point of the Agile Manifesto is to put them firmly in their place.

Comment on this »

Posted at 08:00 on 14 April 2011.
Tagged: agile

Auditing is often seen as mindlessly bureaucratic and something of a waste of time imposed on us by regulators and compliance wonks. However, used effectively, auditing can be a powerful tool.

The purpose of an audit trail is to answer questions. Obviously, it is most effective if (a) it is set up to answer questions that people are actually likely to ask, (b) you are aware of those questions and bear them in mind when you’re actually filling in the form, and (c) your audit trail gives you tools to help you data mine the answers to those questions easily.

Source control is one example. Some people like to set up their source control policies so that every changeset has to be referenced to an item in the issue tracker, so that they can answer the question, “Which changesets fixed this bug?” While this is useful information to have, it isn’t the kind of question that people ask all that often. The kinds of questions people are likely to ask from your source control system are (a) “Which changesets introduced this bug?” and (b) “Who wrote this line of code, and why?”

These are the overriding principles behind how often you should check in code, and it’s also why I’m a great fan of hg bisect.

Change management is another example. Change management is sometimes scorned by some agilists as overly bureaucratic and pointless, but it only becomes that way if it’s implemented without any specific questions in mind. Here are some questions that an effective change management process can answer, for example:

• What services are running on this server?
• Who would know about the services that are running on this server?
• I’ve run into a problem while making a change to this service, has it happened before, and if so how was it fixed?

Most of these questions concern specific resources, such as servers, software, build images, subnets, routers etc. A good change management system will let you tie these specific resources to specific changes, and will let you quickly search through them and get answers to these questions.

Comment on this »

Posted at 08:00 on 7 April 2011.
Tagged: auditing, change management, source control

Long time Mercurial users will no doubt appreciate the new merge and conflict resolution dialogs in TortoiseHg 2.0. When you have some conflicting files, rather than making you go through them one at a time with no idea how many more there are to handle, you are given a list of them with options to help you merge them.

However, there is one feature of this dialog that will no doubt raise an eyebrow or two. Whenever a file has been modified on both sides of the merge, it reports it as a conflict, even if the modifications were to completely different parts of the file. What is going on here? Has Mercurial suddenly forgotten how to merge? Is it turning into Team Foundation Server? Whatever next, read-only files and baseless merges?

Actually, no it hasn’t. That was my reaction when I first tried out the development version of TortoiseHg 2.0 last summer, so I rolled up my sleeves and coded up an option to restore the traditional behaviour:

When you merge, you can also choose on a case by case basis between automatic and manual file resolution:

So why does it work this way now? In my discussions on the TortoiseHg mailing list, Steve Borho, the lead developer of TortoiseHg, pointed out that there’s a lot of hallway usability testing behind it:

I’ll allow that long-time Mercurial users may find this limiting, so I’ve assumed that we’ll eventually add a back door to revert to default Mercurial behavior.  But I have heard from many new users over the years that this is the one part of the Mercurial interface that is unsettling, having kdiff3 thrown at them at seemingly random occasions, so I want the internal:fail approach to be the initial default.

André Sintzoff concurred:

I agree with you. Most of the new users I know are somehow disturbed by the “old” merge behaviour.

When I show them the “new” behaviour, they are enthusiast.

They had a valid point. This is something I’d forgotten about myself.

Inexperienced developers are usually terrified of merging. When you’re combining two people’s changes together, you need to know and understand not only the changes themselves, but the context as well. To delegate the entire process to some unknown computer algorithms sounds reckless and dangerous. This was one of the first things I found intimidating about svn update when I first started using source control in a team context in the first place.

Yet in practice, fully automated merging works remarkably well. When you run svn update or hg merge, more often than not, it all goes very smoothly — in fact, much more so than attempting to merge everything manually. Why should this be?

1. In the overwhelming majority of cases, the default option is the correct one.

Next time you do a merge, turn off automatic conflict resolution and use a three-way tool such as Perforce Merge. I particularly like Perforce Merge because it shows you exactly what’s going on. At the top, you have the two sides of the merge on either side of the original version, so you can tell whether something was added on the left hand side or whether it was deleted on the right hand side:

In the most basic case, automated merge tools assume that if a change was made to one side of the merge, but there is no corresponding change on the other side, that change should be included in the final result. That’s what shows up in the bottom pane. On the other hand, if two people have edited the same part of the file, it shows up as a merge conflict and you have to resolve it manually.

Now here’s the key. Once you’ve carried out a few manual merges, you soon realise that with non-conflicting text differences, you almost never choose anything other than this default option. It becomes evident that working your way manually through a string of differences where you only ever choose the default is largely a waste of time.

2. Manual merges increase the risk of human error.

Having said that, automated merges don’t always get it right, and you do sometimes need to be aware of the context on each side. But — and it is a big but — manual merges fare no better.

Here’s a simple example where both automatic and manual merges are liable to give the wrong result. Let’s say that two developers, Alice and Bob, both make an identical change to a source file on their respective branches — for example, throwing an exception when something can’t be found. Then, Bob commits a subsequent change which backs it out. Should the new code be included in the merge or not?

     Changed ------
    /              \
Original         ???????
    \              /
     Changed - Original

Mercurial and Git both take the line that because Bob undid the change, it should be as if he had never made it in the first place — a feature called “implicit undo” — and therefore, the change should be included in the merge. But that is not necessarily what you want. An ideal version control tool would report this as a conflict, but what happens then?

Here’s what it might look like in your merge tool:

There is no indication whatsoever that as well as being added on the left hand side, that exception was also added on the right hand side and then deleted again. Because your manual merge is a naive three-way merge, with no awareness of history, it also gives you implicit undo, and unless you are particularly on the ball and aware that this change was made then undone in the first place, you won’t pick up on it.

But if you’ve just worked your way through a dozen or more diffs where you’ve chosen the default option every time, the chances are that your eyes will be glazing over, you won’t be on the ball, and you’ll miss it. And therein lies the rub: as well as being slow, manual merge resolution increases the risk of human error.

Another problem with manual merge resolution is that it frequently presents you with diffs that are pretty confusing and overwhelming. Visual Studio .sln files are a particular pain to work with in this respect, since you are dealing with lines and lines of GUIDs that blur into each other. Very often, the only difference between the two sides is that stuff has been moved around. In cases such as these, it can be almost impossible to carry out a manual merge effectively, whereas an automated merge will work out fine. Long lines just compound the problem.

So there’s your trade-off. An automated merge, which may or may not be correct due to ignorance of context. Or a manual merge, which may or may not be correct due to human error and lack of clarity of both context and content. And is several orders of magnitude slower into the bargain.

3. Semantic resolution is easier dealt with by compiling and testing anyway.

The upshot of this is that merging is actually a two-pass process, regardless of how you do it. The mechanical operation of combining your changes is not the be-all and the end-all, but only the first step. Once you’re done with it, you will need to test your merge and fix up any problems. But this isn’t a big deal — it’s the kind of thing you’re doing all the time in normal coding anyway.

Besides, manual resolution only gives you a narrow view of what you’re doing. It’s only when you compile and test that you really see how the two sides of the merge fit together and get a feel for how to deal with the context and intent of the two sides of the merge.

Most problems with merges show up when you attempt to compile your code. In these cases, it’s merely a case of fixing them up — clearing up ambiguous references, checking renames and so on. If you have good test coverage (and you should have good test coverage), your unit tests will pick up the majority of other problems, though you do need to be aware that incorrect merges may have an impact on your tests too. And while some problems may slip through the net, they generally are pretty insignificant in number and scope compared to bugs that creep in through normal, everyday coding.

Fully manual merge resolution is helpful for new users because it eases them gradually into the apparently scary world of branching and merging. But once you are used to it, it becomes apparent that there is little or no benefit to the all-manual approach. While you may feel more in charge of the process while you’re carrying it out, this is largely illusory and a waste of time, somewhat akin to premature optimisation. Provided that your tooling has decent automatic merge support — and Mercurial certainly does have decent automatic merge support — there’s every reason to make the most of it.

Comments (1) »

Posted at 08:00 on 4 April 2011.
Tagged: merging, source control