james mckay dot net

There are two types of programmer in the world.

Now when somebody writes that on their blog, they’re generally criticised as being elitist and condescending. This is because they tend to talk about the “alpha geeks” or the “20%,” and everyone else or the “80%.”

I’m not going to say that. But I am still going to say that there are two types of programmer.

• Those for whom programming is a means to an end.
• Those for whom programming is an end in itself.

If you’ve been following me on Twitter, and if you’ve read my recent blog posts, you’ll know that this has been something of a theme for me for the past couple of months or so. What got me going about it was reading this remark in a blog post by Jeremy Miller a few months back:

I had a great time at CodeMash yet again. For those of us who look at software development as more of a lifestyle than just a way to meet the mortgage, these kinds of events are like a huge dose of nutrition for the soul. I really liked Scott Chacon’s keynote address and it’s got my head a going about how to make our workplace better (see this again).

It’s not a bad post, all in all, but whoa there. Since when was software development supposed to be a lifestyle?

Don’t get me wrong here. I’m all for being passionate about programming, for sharpening the saw, for improving your skillset, for using the best tools for the job, and all that, but when I hear of people describing software development as a lifestyle, I wonder just what makes them tick. And this isn’t a criticism of Jeremy either. It’s more a criticism of myself. There have been times when I myself have slipped into that way of thinking, and I’ve not found it to be beneficial. It’s easy to fall into that trap when you have the kind of mind that enjoys programming, and it takes a good deal of self-discipline to avoid it. In fact, paradoxically, I find that when I get into programming-as-an-end-in-itself mode, I end up getting less code written, not more.

This, in a nutshell, is what’s been behind a lot of the things I’ve been blogging about recently. It’s been behind my recent decision to stop using vim, for instance. For that, and everything else I’ve been looking into, I’ve been asking myself the question: is this a means to an end, or is it an end in itself?

So to those of you who would want to persuade me to reconsider learning vim properly, or to switch from Mercurial to git, please allow me to remind you of one important fact.

You can’t learn everything.

As programmers, we face a firehose of new tools, new technologies, and so on. Many of these are promoted loudly with great fanfare on the Internet by their fans. In the case of some (*cough* git *cough* ruby on rails *cough*) the fanboy hype is deafening. Most of them are claimed by their proponents to be counterexamples to Fred Brooks’ classic essay, No Silver Bullet. Almost none of them are anything of the sort. Yes, they are beneficial to a greater or lesser extent, but in most cases, there are trade-offs.

It simply isn’t possible to give all of them more than a cursory once-over. You have to triage what to learn and what not to learn. Aggressively. Learning vim would rob me of time familiarising myself with other, more important technologies such as Selenium or backbone.js, for instance. Frameworks and tools that you do learn are huge and complex these days, and if you try to learn too many of them, you end up with a very superficial understanding of them that can lead to problems. Either that, or else you end up spending so much time on the computer that you damage your health and your relationships with other people.

So in the past few weeks, I’ve been taking a good hard look at everything I’m doing, re-evaluating what I should be taking up, what I should be dropping, and what I should just allow to carry on as-is. With that in mind, I’ve been formulating some criteria by which I should decide what to adopt, and what to ignore. With everything I’ve been doing, I’m asking myself a few pertinent questions now:

• Does this solve a problem that I am currently facing or am likely to face in the not too distant future? If so, how effectively?
• Will knowing it enhance my career prospects? More importantly, will not knowing it harm my career prospects?
• Are there any examples of real-world applications using it, or is it all smoke and mirrors?
• What is the ramp-up time, and is it worth it?
• Is it in line with the direction I am heading, or is it a distraction?
• If it makes programming easier, does it do so at the expense of the end users?

The last point is a key implication of programming being a means to an end rather than an end in itself. If programming is a means to an end, and you care about your end users, you will base your choice of tools and frameworks on what serves them best. On the other hand, if programming is an end in itself, you will choose the cool languages and tools that everyone is raving about, but at the expense of your audience. This is of course what WordPress got right but Diaspora got wrong.

I’ve decided that in terms of tools and technologies, I should focus on my core competencies: namely, the Microsoft .NET stack, HTML 5 and JavaScript. I also want to up my game as far as test-driven development is concerned — hence, I’ll probably have another, closer look at MSpec — and also get to grips much more with agile development methodologies, in particular Scrum, which we have been using at work for the past year or so. As far as my other skills are concerned, such as PHP, Python, Django and Linux, these are useful skills to have, but how much time I spend on them from now on is something I am yet to decide. While it’s good to diversify to an extent, I don’t think much of the Pragmatic Programmers’ suggestion that you should learn another language every year, simply because that takes you into the realm of programming for programming’s sake, and besides, I don’t think it’s realistic, unless you are content to be a jack of all trades and master of none.

Bloggingggaaaaaaaa!

The ALT.NET movement makes me feel a little bit uneasy somehow. While there are some pretty good people in there, and I fully support the aim of finding better ways to do things, the movement as a whole seems to have largely degenerated into a clique of programming-as-an-end-in-itself Ruby fanboys with inflated egos who only do .NET under protest for their day jobs, and who complain about Microsoft tools purely for the sake of complaining about Microsoft tools. I ended up in conversation on Twitter with several of them in the wake of my TFS==Lotus Notes blog post, and I think some of that may have rubbed off onto me. Certainly, I’ve been doing far too much complaining recently, and I’m sure it doesn’t make me look good.

I’ve come to the conclusion that the programmers most worth following on Twitter, most worth subscribing to in Google Reader, are the people who are building end-user applications aimed at non-developers. This isn’t necessarily because they’re smarter — frameworks and developer tools are technically more challenging beasts to write and maintain — but because they tend to have a better grip on reality. Building end-user applications puts activities such as socialising and speaking to non-programmers right at the heart of programming: you have to talk to people before you start in order to get ideas, and then you have to talk to them afterwards in order to make sure you’ve delivered something that they can use without getting confused. And of course, you also have to talk to them in between times to make sure you are on the right lines. This is where agile software development hits the nail on the head — developers are expected to talk to the customers on a daily basis. On the other hand, only ever building programming tools and libraries, and little or nothing else, seems a bit self-serving to me.

At the same time, I’ve also been asking the same questions about my blog. Over the past few years, I’ve constantly found it to be a very time consuming animal, where I don’t see much benefit coming out of it, and I keep wondering why I do it. Most of my recent blog posts have said, “You should be doing this,” or “You shouldn’t be doing that,” or “This is a cool tool,” or “That process is bad,” or other things in that vein, and I’ve been saying very little about what I’ve actually been doing myself. This is just punditry, and I don’t want to be just another talking head. We have too many talking heads in our industry, and it gets rather boring after a while.

I also think I got carried away somewhat with the whole Mercurial/source control thing. Now don’t get me wrong: I still maintain that Mercurial is the best source control tool available today by a long shot, that only git comes anywhere near it, that the old-school centralised model is past its sell-by date and no longer fit for purpose, and that if you’re not using a DVCS, you’re harming your productivity, making a rod for your back, and exposing your project to unnecessary and unacceptable risks. But there comes a time when you have to let things drop, otherwise you end up coming across like a noisy fanboy yourself and you just annoy people who have missed them as they flew straight past coming out of the firehose. In fact, this fanboy attitude is totally out of order in the Mercurial culture itself, where viewing programming (and by extension, source control) as a means to an end and not an end in itself is very much the norm.

I’ve therefore decided that I need to take a break from blogging for the next few months. Quite when, or even whether, I return is anyone’s guess, but if I do, I’m not planning to carry on writing in the same vein as I’ve been doing so up to now. If I’m to carry on blogging, I want to build stuff that I can talk about and showcase here on my blog first. Unfortunately, at the moment, most of the code I write that’s worth talking about is at work, and there’s obviously a limit to how much I can write about that. But the Agile Manifesto tells us to favour working software over comprehensive documentation, and as a corollary, the contents of your github or bitbucket account say more about you as a developer than the contents of your blog.

I’ll no doubt continue working on some personal programming projects in my spare time. In fact, hopefully I’ll be able to spend more time working on programming projects without getting distracted by blogging about x, y or z. But it will have to be time-boxed anyway. During the week, I am out of the house from seven in the morning until seven in the evening, which doesn’t leave me much time for anything other than sleep, and I want to free up more time for socialising, church activities, and developing other interests that don’t involve sitting over a computer.

Comments closed (2) »

Posted at 08:00 on 3 October 2011.
Tagged: programming

Here are a few of my random thoughts on Ayende’s oh so controversial post, If you don’t have pet projects, I don’t think I want you:

1. Pet projects don’t need to be extensive or advanced for you to stand out from the crowd. The vast majority of .NET developers have no personal projects whatsoever, so as little as one evening a month spent on them should be more than adequate.

2. Can we cut out all the cringeworthy Bravo Sierra about “passionate developers,” please? There are less clichéd ways of saying much the same thing, and besides, I get the impression that some recruiters view passion as a substitute for competence. It isn’t. You get plenty of passionate developers who think that Pokémon exception handling is perfectly acceptable, for instance.

3. It’s also possible to have too much passion for programming. People who get too passionate about programming forget that it is a means to an end, and not an end in itself. A telltale sign that you are dealing with such a person is that their github account contains nothing but vim scripts and clones of developer tools and class libraries. If, as a recruiter, you want to look for personal projects, look for ones aimed at non-developers. That way, you get some indication that you’re dealing with someone who at least has some grip on reality.

Comments closed (1) »

Posted at 08:00 on 15 September 2011.
Tagged: programming

About a month or so ago, elite C# gurus such as Eric Lippert and Jon Skeet were discussing the new async and await keywords that are forthcoming in C# 5. Eric Lippert introduced them with a series of blog posts on continuation passing style programming that I haven’t yet got round to reading properly.

People who are into continuation passing tend to wax lyrical about it, portraying it as the latest cure for cancer. But I’m wondering just how much of my time I should spend on it. To me, it looks a bit whacked out — isn’t it just some kind of nonlocal goto, and aren’t goto statements considered harmful? How will it help me write code to keep the Great British Public informed about what their elected representatives are discussing in Select Committees?

You never know. No doubt in a few years’ time, I too will be passing continuations around like sweets at a children’s party. I had similar thoughts a few years back when I first read about LINQ. Why would anyone want to embed SQL statements in their C# code, I thought? It sounded a bit like the bad old days of tag soup in classic ASP. But nowadays I use LINQ all the time and I’ve long since figured that LINQ is not LINQ to SQL. In fact, I don’t see how you can get anything done without LINQ. It’s just so much cleaner than writing a whole bunch of for loops. And it gives you lazy evaluation, which can give you massive performance benefits if you use it properly.

This is a common paradox in programming. Once you get to grips with a new programming language feature, concept or tool, you wonder how on earth you managed to write any code at all without it. But on the other hand, when you look at all the concepts, features and tools that people are waxing lyrical about in those terms, they all look weird, if not completely scary. And when you wax lyrical about something or other, as sure as eggs are eggs, someone will tell you your tool is the weird and scary one.

In his essay, “Beating the Averages,” Paul Graham calls this the Blub Paradox:

Blub falls right in the middle of the abstractness continuum. It is not the most powerful language, but it is more powerful than Cobol or machine language.

And in fact, our hypothetical Blub programmer wouldn’t use either of them. Of course he wouldn’t program in machine language. That’s what compilers are for. And as for Cobol, he doesn’t know how anyone can get anything done with it. It doesn’t even have x (Blub feature of your choice).

As long as our hypothetical Blub programmer is looking down the power continuum, he knows he’s looking down. Languages less powerful than Blub are obviously less powerful, because they’re missing some feature he’s used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn’t realize he’s looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.

When we switch to the point of view of a programmer using any of the languages higher up the power continuum, however, we find that he in turn looks down upon Blub. How can you get anything done in Blub? It doesn’t even have y.

I guess the best definition of “Blub” is the collection of programming language features, concepts and tools that you know and understand — or at least, that you can see might come in useful, so you really ought to get round to learning about them. It’s what makes sense from your vantage point. But when you look up and see someone waxing lyrical about something unfamiliar, don’t just write it off as weird. They may quite possibly be right.

Comments closed »

Posted at 08:00 on 13 December 2010.
Tagged: jargon, programming

The question of the command line is a controversial one, with some people declaring that it should be a core competency without which you should be considered an automatic “no hire,” and others claiming that it’s an anachronistic irrelevance in these days of graphical user interfaces, drag and drop, and integrated everything in Eclipse and Visual Studio.

Now I’ve repeatedly described command line instructions as a bad marketing strategy, and I stand by that. The command line is pretty intimidating if you aren’t already familiar with it: it has a much steeper learning curve than GUI tools, it lacks discoverability, and it only gives you a narrow view of what you’re doing that can seem like peering down the wrong end of a telescope at first. Consequently, there are also some tasks for which it is simply not suited. Furthermore, its support on Windows is abysmal.

But these sentiments of mine are only about marketing, and do not extend to day-to-day use. Over the years, I’ve actually come to appreciate the benefits of the command line. This is probably because I did a lot of work with Linux at my last job, so I had to get to grips with it. (One will recall that I was thrown in right at the deep end by being told to install Gentoo on a spare server.) And in fact, once you’re familiar with it, it offers some significant advantages for many tasks. These are they.

1. Ergonomics and speed.

The command line is keyboard-centric. GUI tools, on the other hand, are mouse-centric.

Mouse-centric environments have a nasty habit of forcing you to reach to and fro between your keyboard and mouse the whole time, which not only slows you down considerably, but can also cause repetitive strain injury. The fact that your keyboard puts a numeric keypad in the way doesn’t help much, and with some otherwise ergonomic keyboards such as the Microsoft Natural 4000, the stretch can be horrendous. That’s why many power users prefer command line instructions and keyboard-focused text editors such as emacs or vim: your hands stay in one place and let you work a lot more quickly and fluently.

Distraction is another factor. When you have to move your hand between the keyboard and the mouse all the time, it can be tempting to focus on mouse-centric tasks such as sorting out your e-mail inbox and surfing the web. On the other hand, when you’re spending most of your time on the keyboard, it’s much easier to stay focused on what you have to do without getting distracted.

2. Scriptability.

Most command-line haters think nothing of writing a long, complex howto document outlining the instructions needed to carry out a particular repetitive task, such as setting up a new project in source control, or building your project, or adding a new component from a template. The only problem with this is that when someone new comes onto the team, they have to follow these instructions through step by step, and the more steps there are to follow through, the more likely they are to misunderstand them, get one of them wrong or out of order, and end up deleting all their files by mistake.

You could write a PowerShell script to automate it — provided, of course, that the tool concerned actually exposes enough functionality through PowerShell or the command line to let you do so — but they generally don’t because that would require going to the trouble of learning a whole new language for something that you don’t do very often. But on the other hand, if you’re used to using the command line (or PowerShell) all the time in the first place, you already have a suitable language at your fingertips, and you don’t need to learn anything new. Scripting your repetitive tasks becomes the easy, logical and natural thing to do.

3. Cut and paste.

Command line instructions are much easier than screenshots to copy and paste into an e-mail, or a howto guide, or a blog post, or an instant messaging conversation. While this is inappropriate for introductory tutorials, it can save a lot of time when you need to Get Things Done.

For the past few days I’ve had to work from home because of the snow. At one point, I had to give a colleague some instructions on how to do one or two simple things with Mercurial. If we had only had TortoiseHg, I would have had to set up a remote assistance session with him, which would have been overkill for what he needed to do. However, with the command line, I was able to simply type the necessary command line instructions straight into Microsoft Office Communicator, and it was all very simple, quick and painless.

In conclusion.

The command line is not a golden hammer. There are some tasks for which it is not well suited, or which it makes considerably more difficult. But for a lot of tasks, it can provide significant benefits. Agility with the command line can still be a useful skill in a developer’s toolkit, and it is quite wrong to write it off completely.

Comments closed (1) »

Posted at 08:00 on 6 December 2010.
Tagged: programming, tools

This is the first in an irregular series of blog posts about jargon that you’ll find from time to time being bandied about by programmers who read blogs, such as my good self.

I’ll start off with a tale first told by the physicist Richard Feynman. The story goes that during the Second World War, various Pacific islands were invaded first by the Japanese then by the Americans. They set up Air Force bases, with runways and control towers and ground traffic controllers with headphones and batons to direct the aeroplanes to where they had to park up and drop off supplies. The natives were amazed: they had never seen anything like it.

Eventually, the Second World War came to an end, and the Japanese and the Americans packed up and went home. The natives, however, built their own runways, bamboo control towers and large bamboo mock-ups of aeroplanes, and stood wearing bamboo headphones and waving bamboo sticks in a vain attempt to summon the aeroplanes from the sky with their cargo. Needless to say, no planes came. They had copied the form of what the Americans were doing, but they didn’t understand why they were doing it.

Sounds far fetched? Apparently it’s a true story, and some of these cargo cults are still in existence today.

In a programming context, the term “cargo cult” is used to refer to someone doing something without properly considering why they’re doing it, or whether what they’re doing is even appropriate. Blindly copying and pasting code from tutorials and code samples on the web is one example. If you’ve ever seen something like this:

void Page_Load(object sender, EventArgs e)
{
    try {
        DoSomething();
    }
    catch (Exception ex) {
        Console.WriteLine(ex.ToString());
    }
}

and wondered what that Console.WriteLine statement is doing in a web application, which doesn’t even have a Console to WriteLine to, it’s because someone has cargo culted something over from a console application in a code sample on MSDN. Another example is claiming that stored procedures are the answer to SQL injection vulnerabilities, when in actual fact it’s all to easy to subvert this by smashing strings together in your stored procedure and exec-ing the result:

create procedure ListStudents(@OrderBy varchar(100)) as
begin
    declare @sql varchar(200)
    set @sql = 'select * from Students order by ' + @OrderBy
    exec @sql
end

One thing about cargo culting is that we all do it at some stage or another when we’re getting to grips with a new and unfamiliar technology. This is why code reviews are such a good idea: someone more knowledgeable about the technology concerned will be able to pick up on mistakes such as these and fill in the gaps in our knowledge. But the important thing is that we should always ask why we’re doing what we’re doing, and not just stick around indefinitely at the cargo cult level.

Comments closed »

Posted at 08:00 on 18 November 2010.
Tagged: jargon, programming

Getting worked up about tabs versus spaces and consistent indentation may sound completely over-the-top OCD to some people, but there are very good reasons for it. It doesn’t matter too much whether you prefer tabs or spaces, but you must never mix the two in your code. Here’s why.

Inconsistent indentation is a menace when you’re trying to see what your code does.

The reason why we indent code in the first place is so that we can see unambiguously, at a glance, where a block of code, such as an if statement or an HTML <div> tag, begins and ends. It helps us to speed-read code and zoom out from the details to the higher level design with less cognitive overload. Without clear, consistent, unambiguous indentation, it’s easy to end up getting your code muddled up and introducing bugs.

Another problem is that different editors and other tools treat the tab character within your file in different ways. Some of them display it as four spaces, some of them as eight, and some people configure it to display as something completely different altogether. If you’re mixing tabs and spaces within the same function, it can look really confusing if for some reason someone has to load it up into an editor or other tool that behaves differently from what you’re normally used to. Such as, for instance, your source control’s diff/merge tool, or a web-based system such as Trac.

Inconsistent indentation is a menace when you’re merging changes.

If your indentation is inconsistent, sometimes my only option is to reformat it just to make sense of what’s going on. Even when I don’t, the chances are high that Visual Studio will do it for me. But this causes other problems. When your source control tool merges changes, whether you’re using svn update or hg merge or whatever, it does so by comparing your code on a line-by-line basis. If you change a line from tabs to spaces or vice versa, or increase or decrease the indentation by so much as a single space, it views it as deleting one line and replacing it with another. Consequently if you’re changing between tabs and spaces all over the place, you’re going to be setting yourself up for Big Scary Merges.

Most visual diff tools give you the option to ignore whitespace when you’re comparing files, but it still makes it much more difficult to carry out the merge, since you will get more conflicts when you attempt to merge the changes automatically.

Always turn on visible white space.

The first thing I do whenever I install a new editor is to turn on visible whitespace, and I always recommend other people to do the same, so you don’t miss monstrosities such as this:

Some people object to this, saying that all the extra dots and arrows makes their source code look cluttered. This is a valid point — Visual Studio’s default colour scheme does make it look rather intrusive. If you find this to be a problem, try changing the colour of your visible whitespace to something a bit lighter. You can get away with it being quite faint: just as long as it is enough to bring to your attention any inconsistency between tabs and spaces here, or trailing whitespace at the end of a line:

Even so, working with visible whitespace can take a bit of getting used to. But get used to it you must. Otherwise you’re just going to cause confusion and make life difficult not only for yourself but also for anyone else who has to work with your code.

Comments closed (3) »

Posted at 08:00 on 15 November 2010.
Tagged: programming

Half a dozen or so of us from work were at the London Stack Overflow Dev Days event with several hundred other developers today. I’ve been pretty impressed with the way Jeff Atwood and Joel Spolsky’s enterprise has turned out to be such a resounding success, and I’ve also been an avid reader of Jeff’s blog, Coding Horror, for several years now, so I was naturally delighted to get the opportunity to go.

Encountering Joel and Jeff in real life was an interesting experience, since I’ve only ever read their blogs and Twitter feeds up to now. Over the past year or so, I’ve had to get used to seeing certain people in real life that most people only ever see on TV or on the Internet, but it still seems a bit odd when you do. It certainly gives you a totally different impression of them from what you had before though. You can certainly see why Joel and Jeff in particular are both so successful in what they do: as well as being excellent online communicators, they are both brilliantly engaging and entertaining public speakers. So too was Jon Skeet, who gave a very funny talk about localisation entitled “Humanity: Epic Fail,” assisted by a sock puppet called Tony the Pony. Joel’s talk on FogBugz was a pretty hard sell, but it certainly looks impressive, boasting a feature set that makes Trac look like Notepad.

The other talks included an introduction to Python by Michael Sparks of the BBC, who explained to us Peter Norvig’s 21 line spelling corrector (it didn’t escape my attention that Jon Skeet spent the lunch break porting it to C#); introductions to mobile development for no less than three rival platforms (Google Android by Reto Meir, iPhone by Phil Nash, and Qt/Nokia by Pekka Kosonen); introductions to jQuery (Remy Sharp) and Yahoo! Developer Tools (Christian Heilmann); and an academic talk on “How not to design a scripting language” by Paul Biggar, who recommended the book “Engineering a Compiler,” by Cooper and Torczon as a superior alternative to the Dragon Book.

I spoke to Jeff during the afternoon break and asked him if he had any plans to publish the best of Coding Horror in a book. He said he’d thought about it a bit, but wasn’t entirely convinced it was worth doing. It’s something I’ve recently thought that he’d do well to do—a lot of his posts are ones I’d consider “must-reads” for every working developer, and if he did, I’d buy it in a shot. He wouldn’t be the first person to do something like that either—after all, Joel did it (twice), and so did Raymond Chen. It was interesting what he asked me when I told him I work for Parliament—he was most interested to know whether Britain is part of Europe or not. It’s a good question, that. Officially we are, but unofficially I sometimes think that as a country, we’re not entirely sure ourselves.

There were just two disappointments to the day. One was the catering. I was half expecting something along the lines of a buffet lunch—after all, I do tend to think of the Fog Creek Way as one where they go the extra mile to get these things perfect—but it turned out to be the kind of mass produced sandwiches that you get in a motorway service station that are all ridiculously overpriced, taste exactly the same as each other, and don’t meet with my approval anyway because they’re spread up with margarine. The other disappointment was the venue itself. Kensington town hall simply is not big enough for however many of us (800? 1000?) were there today. Consequently it felt very crowded and claustrophobic, and even a little bit uncomfortable, especially during the breaks when we all crowded into the foyer and had to form a queue stretching seemingly all the way to Barking and back to get to the food.

The day ended at about ten past six and I came away with a whole lot of freebies: a Qt rucksack, a copy of the Aardvark’d DVD, a handful of FogBugz pens, and a handful of Stack Overflow, Server Fault and Superuser stickers. All in all, it was a pretty full day (I had to get up half an hour earlier than usual and I got home an hour and a half later than usual, and sitting through seven hours of talks was pretty intense) but it was well worth it.

Comments closed »

Posted at 23:00 on 28 October 2009.
Tagged: programming

There seems to be a bit of intellectual snobbery among some non-web developers, who regard web development as a soft skill, something other than software engineering that’s only for programming wusses who can’t make the grade to get into desktop development. It’s an attitude that is typified by this flabbergastingly arrogant post by Michael Braude (hat tip: Jeff Atwood) who has this to say:

But then, that’s just it, isn’t it?  The reason most people want to program for the web is that they’re not smart enough to do anything else.  They don’t understand compilers, concurrency, 3D or class inheritance.  They haven’t got a clue why I’d use an interface or an abstract class.  They don’t understand: virtual methods, pointers, references, garbage collection, finalizers, pass-by-reference vs. pass-by-value, virtual C++ destructors, or the differences between C# structs and classes.  They also know nothing about process.  Waterfall?  Spiral?  Agile?  Forget it.  They’ve never seen a requirements document, they’ve never written a design document, they’ve never drawn a UML diagram, and they haven’t even heard of a sequence diagram.

Well I have news for you, Michael. This is just plain wrong. As web developers, we have to know and understand almost everything you’ve listed above—and more. Practically the only exceptions that he has listed are compilers, pointers and virtual C++ destructors, and even then some of us may need to get to grips with them from time to time. We need to understand class inheritance, interfaces, abstract classes, virtual methods, pass by reference versus pass by value, garbage collection and so on or we’re toast.

In fact, some of the concepts he’s listed above are even more critical to the web than other, supposedly more superior, forms of development. Take concurrency for instance. This is a particularly difficult concept to understand, debug, work with and test, and on desktop applications which only ever run in single user mode, it is very often a non-issue. You can build desktop applications for years without needing to know jack squat about concurrency, but once you start building web applications, which are multi-user by their very nature, it’s only a matter of time before you get bitten by it.

There are other aspects to web development that we need to understand to a much greater extent than desktop application developers. Security is much more important, for instance. It isn’t exactly a non-issue for desktop applications—web browsers, Microsoft Anything, and so on can be potential attack vectors, especially if they allow scripting—but web servers and web applications are low hanging fruit as far as hackers are concerned, and you have to contend with bots actively probing your site, 24/7, for data injection and cross site scripting vulnerabilities.

I could give other examples. We have to understand scalability and caching. We have to understand not only HTML but also the protocols that run the web, such as HTTP, TCP/IP, SMTP, SOAP, and so on. We have to work with several different languages at once—HTML, JavaScript, PHP or C#, SQL, CSS, and XML and its many domain-specific variants.

We need to work in cross-browser environments that very often make test-driven development particularly tricky if not impossible. (How do you unit test CSS positioning, for instance?) We need to understand graphic design, accessibility, and usability. We certainly do need to understand business processes such as agile, waterfall, scrum, and so on. We need to understand UML. Most difficult of all, we need to know how to bridge the gap between the exact literalism of computers and the vagaries of humans and other living beings.

And guess what? Far from being a cuddly toy, JavaScript is a rich, powerful, functional language that, despite its C-style syntax, was originally heavily influenced by Scheme.

To be sure, there are a lot of incompetent web developers out there, and the barrier to entry is ridiculously low, but it is thoroughly incorrect and misleading to say that this is “because you don’t need to know complicated things to be a web developer.” On the contrary, if you’re working on a site of any significant proportions, you need to know some pretty hard stuff. You need to be pretty smart to be a successful web developer.

Comments closed (3) »

Posted at 08:00 on 17 August 2009.
Tagged: programming

There is a corollary to the concept of technical bankruptcy that I wrote about last week. When you take over a project that was originally built by another company, there will obviously be the inevitable ramp-up time on top of that, as your developers get to grips with the system, but on top of that, you are taking on that project’s technical debt.

We learned this the hard way with one of the worst projects I ever worked on. The client needed a new front end to their website in a timescale of only six weeks, and the sales guys said that of course we could do it, as non-technical sales guys generally do. However, only when they had signed on the dotted line did we get to look at the code.

The levels of technical debt were horrendous. The system, which had little or no coherent documentation, consisted of dozens of classic ASP files of extremely ugly copy-and-paste code, two thousand line stored procedures with up to eighty parameters each, and I’m sure that some of the identifier names were in Klingon. Five months of late nights, much stress and one ruined Christmas later, we eventually delivered something that I considered approximately halfway between “me-ware” and “pre-alpha” in quality, then spent the next two months frantically fixing bugs in the system.

There is a cautionary tale here. When you are taking over a system that was developed by another company, there is absolutely no way that you can give any estimates whatsoever until you have looked at the code base. A five-stage booking form may not sound like a big deal, but when you have to plug it into convoluted and undocumented spaghetti code that borders on the incomprehensible, all bets are off. In this case, the business logic was all in the two thousand line stored procedures, which were designed entirely with the workflow of the old site in mind, and the workflow of the new site was substantially different. On top of that, the client also wanted web services integration, so we had to build a fairly complex abstraction layer on top of it.

In any case, you should certainly ask why the previous company isn’t supporting it any more. It may be that the technical debt grew to levels that they can no longer handle, and that is why they are fobbing it off onto you.

Comments closed »

Posted at 08:00 on 14 April 2009.
Tagged: programming

When clients and managers insist on a quick and dirty fix, they usually think that this means we only implement the “must-have” functionality, and leave aside the “nice-to-have” and even some of the “should-have” stuff. Okay, so we might need to reboot the server once a month or so, or we might need to go directly into the database using SQL Server Management Studio whenever prices change, but we can live with that, they think.

However, there is an unseen and potentially much more damaging effect of the quick and dirty fix, that developers, managers and clients all need to be aware of: technical debt.

Martin Fowler explains it like this:

You have a piece of functionality that you need to add to your system. You see two ways to do it, one is quick to do but is messy – you are sure that it will make further changes harder in the future.The other results in a cleaner design, but will take longer to put in place.

Technical Debt is a wonderful metaphor developed by Ward Cunningham to help us think about this problem. In this metaphor, doing things the quick and dirty way sets us up with a technical debt, which is similar to a financial debt. Like a financial debt, the technical debt incurs interest payments, which come in the form of the extra effort that we have to do in future development because of the quick and dirty design choice. We can choose to continue paying the interest, or we can pay down the principal by refactoring the quick and dirty design into the better design. Although it costs to pay down the principal, we gain by reduced interest payments in the future.

Now there are times when technical debt is an appropriate strategic decision. Often, you need to get to market quickly with your website in order to obtain a competitive advantage. Sometimes, there are hard deadlines to meet. However, just as with financial debt, you need to be aware that you are taking on a certain amount of technical debt, because it will need to be repaid sooner or later.

Consequently, when you make the strategic decision to accrue some technical debt, you should always allow for some time to repay it at regular intervals — refactoring, adding more unit tests, documenting the system, and maybe even rewriting some components from scratch a second time if they are particularly messy. If you don’t, and the situation is allowed to get out of hand, you can end up with technical bankruptcy.

Technical Bankruptcy is the point at which levels of technical debt become so overbearing that the project is effectively doomed. This is due to two serious and crippling effects of technical debt:

1. Making changes becomes much harder.

One of the quickest ways to build up technical debt is to copy and paste large sections of your code. Programming purists rightly regard copy and paste as evil — it violates the “once and only once” rule and makes it much harder to make changes later on. However, it can add extra functionality into your program much more quickly than doing things the “correct” way. Many of the other programming anti-patterns, such as magic numbers, meaningless method names such as “do_it”, and so on, fall into the same category — shortcuts which involve a trade-off between getting things done now and getting things done in the future.

However, these are the less complex cases. More serious are the higher-level shortcuts: poorly designed database schemas, tight coupling between the tiers of the application, a lack of preliminary analysis and design, and so on. Nevertheless, whatever the cause, the result is that by asking for quick and dirty hacks today, you are potentially blocking yourself off from asking for quick and dirty hacks tomorrow.

2. Finding developers to work on the project becomes much harder.

Make no mistake about it: working on a project that has an excessive amount of technical debt is a depressing, discouraging, and disheartening experience. It can mean going for days or even weeks on end banging on the keyboard with little to show for it and getting shouted at by your boss for not delivering on time. It saps morale and by extension productivity, and this can spill over onto other projects and pull the rest of the team down. And what do developers do in that kind of situation? If they’re any good, they go and find somewhere else to work.

This compounds the problem even more. When a developer leaves, you have to recruit someone else to fill their place. This means placing an ad wherever you normally place ads, spending time filtering CVs, conducting phone screens, technical tests (you do give potential hires a technical test of some nature, don’t you?) and interviews, and then once you find a suitable candidate, there is the ramp-up time where they are actually getting to grips with the system — and no prizes for guessing what technical debt does to the length of this ramp-up time.

Although we are in the throes of a recession, there is still a massive shortage of good developers. It’s amazing just how many working programmers there are out there who can’t even handle the most basic challenges such as the FizzBuzz problem, and who get by merely by copying and pasting SQL injection vulnerabilities from ten year old PHP tutorials, or by pestering their more talented and knowledgeable colleagues for help at every step of the way and proving to be a net drain on the company’s resources. These are the kind of developers who have no concept of technical debt in the first place, and will just make things worse faster.

I would say that when you start to find it difficult to get good developers to work on a project for any length of time, it is a sign that your project is in serious trouble.

Is a total rewrite the answer?

Managers are invariably suspicious whenever a developer recommends a complete rewrite. Joel Spolsky wrote an article a while ago asserting that complete rewrites are a bad idea, and every time I say that X needs to be rewritten, this article is quoted back to me as a stock excuse for declining my suggestion.

Now the managers have a point here. A total rewrite from the ground up is the nuclear option: you run the risk of throwing away a whole lot of undocumented business rules, corner cases and bug fixes, and a complete redesign of the administrative interface means staff re-training and a deluge of support calls. Besides, developers can be a bit trigger-happy when calling for a total rewrite. We tend to be a pretty opinionated bunch, and very often we will call Someone Else’s Code complete garbage simply because we don’t understand it properly. But given a bit of time, most developers will manage to settle down into a new code base and will be able to start improving it and cleaning it up by a process of small, incremental refactorings as they go along.

Nevertheless, once you reach the point of technical bankruptcy (where technical debt becomes so overwhelming that the project is effectively doomed), a total rewrite might save the day, and maybe even provide some added value by clearing out certain business rules and systems that were put in place only as workarounds for bad design decisions in the first place. However, it would take a brilliant and determined developer, probably working on it as a skunk works project, to pull it off. Best to avoid getting into the situation in the first place, by paying down your technical debt sooner rather than later, and only hiring developers who know what they are doing to work on the project.

Comments closed (3) »

Posted at 08:00 on 6 April 2009.
Tagged: programming