james mckay dot net

Blah blah scribble scribble waffle waffle
24
Jul

Interchangeable data access layers == stealing from your client

This is one of those so-called “best practices” that crops up a lot in the .net world:

You need to keep the different layers of your application loosely coupled with a clean separation of concerns so that you can swap out your data access layer for a different technology if you need to.

It all sounds right, doesn’t it? Separation of concerns, loose coupling…very clean, SOLID, and Uncle Bob-compliant, right?

Wrong. Contrary to what you may have been told by .net traditionalists, this is not a best practice. Oren Eini, aka Ayende Rahien, debunked it thoroughly about four years ago.

What he said needed to be said. Trying to abstract out your data access layer so you can swap it out is an exercise in chasing rainbows: it just piles on friction and obstructs you from other, more important optimisations. Every project that I’ve ever seen that has attempted this kind of abstraction has been riddled with severe SELECT n+1 problems that could not be resolved without breaking encapsulation.

Nobody is saying here that separation of concerns is bad per se. What is bad, however, is inappropriate separation of concerns — an attempt to decouple parts of your system that don’t lend themselves to being decoupled. Kent Beck has a pretty good guideline as to when separation of concerns is appropriate and when it isn’t: you should be dealing with two parts of your system which you can reason about independently.

You can not reason about your business layer, your presentation layer, and your data access layer independently. User stories that require related changes right across all your layers are very, very common.

(Nitpickers’ corner: I’m not talking about test mocks here. That’s different. It’s relatively easy to make your test mocks behave like Entity Framework. It’s orders of magnitude harder to make NHibernate or RavenDB behave like Entity Framework.)

Because of this, implementing the strict separation of concerns necessary to make your persistence mechanism pluggable means enforcing strict protocols which slow down day to day development, increase your estimates significantly, adversely affect performance, and make some user stories impossible — all for a requirement that 99% of projects never need, and even for those that do, you can’t guarantee that it will even work, that you won’t have overlooked a myriad of very subtle issues that trip you up all over the place.

You will be diverting significant amounts of developer time and resources — for which your client is paying — away from adding business value which they are asking for, onto a high-maintenance non-functional requirement which they are not.

In other words, if you are attempting to make your data access layer interchangeable, you are stealing from your client.

If you can present a valid business case for making your persistence mechanism interchangeable, then it’s a different matter, of course. But even then, you should still warn them of the extra costs involved as well as the risk that it may not even be possible in the end. Otherwise you won’t be delivering good value for money.

16
Jul

The Anaemic Business Layer

The three-layer architecture, with your presentation layer, your business layer and your data access layer, is a staple of traditional .net applications, being heavily promoted on sites such as MSDN, CodeProject and the ASP.NET forums. Its advantage is that it is a fairly canonical way of doing things, so (in theory at least) when you get a new developer on the team, they should have no trouble in finding where everything is.

Its disadvantage is that it tends to breed certain antipatterns that crop up over and over and over again. One such antipattern is what I call the Anaemic Business Layer.

The Anaemic Business Layer is a close cousin of the Anaemic Domain Model, and often appears hand in hand with it. It is characterised by business “logic” classes that don’t actually have any logic in them at all, but only shunt data between the domain model returned from your ORM and a set of identical model classes with identical method signatures in a different namespace. Sometimes it may wrap all the calls to your repository in catch-log-throw blocks, which is another antipattern in itself, but that’s a rant for another time.

The problem with the Anaemic Business Layer is that it makes your code much more time consuming and difficult to maintain, since you have to drill down through more classes just to figure out what is going on, and you have to edit more files to make a single change. This in turn increases risk because it’s all too easy to overlook one of the places where you have to make a change. It also makes things restrictive, because you lose access to certain advanced features of your ORM such as lazy loading, query shaping, transaction management, cross-cutting concerns or concurrency control, that can only properly be handled in the business layer.

The Anaemic Business Layer is usually symptomatic of an over-strict and inflexible insistence on a “proper” layered architecture, where your UI is only allowed to talk to your business layer and your business layer is only allowed to talk to your data access layer. You could make an argument for the need for encapsulation — so that you can easily change the implementation of the methods in the business layer if need be — but that’s only really important if you’re producing an API for public consumption by the rest of the world. Your app is not the rest of the world, and besides, those specific changes tend not to happen (especially for basic CRUD operations), so I’d be inclined to call YAGNI on that one.

The other reason why you might have an Anaemic Business Layer is that you’ve got too much going on in your controllers or your data access layer. You shouldn’t have any business logic in either, as that hinders testability, especially if you’re of the school of thought that says your unit tests shouldn’t hit the database. But if that’s not the case, then it’s time to stop being so pedantic. An Anaemic Business Layer serves no purpose other than to get in the way and slow you down. So ditch your unhelpful faux-“best practices,” bypass it altogether, and go straight from your UI to your repository.

10
Jul

On dark matter developers and the role of GitHub in hiring

The term “dark matter developer” was coined by Scott Hanselman a couple of years ago, when he wrote this:

My coworker Damian Edwards and I hypothesize that there is another kind of developer than the ones we meet all the time. We call them Dark Matter Developers. They don’t read a lot of blogs, they never write blogs, they don’t go to user groups, they don’t tweet or facebook, and you don’t often see them at large conferences. Where are these dark matter developers online?

The problem with Scott’s post is that he doesn’t give a very clear definition of dark matter developers. Certainly, I get the impression that a lot of people confuse dark matter developers with 501 developers, or low-end developers, or technological conservatives. Take a look at this Hacker News thread for instance—some people seemed to be categorising themselves as dark matter developers even though they were actively contributing to open source projects.

Here’s my suggestion for a more precise definition of a dark matter developer:

A dark matter developer is someone who has not made any evidence publicly available that they are able to do what their CV and LinkedIn profile claim that they can do.

This is pretty much the point that Troy Hunt makes in his blog post, “The ghost who codes: how anonymity is killing your programming career.” To be sure, he does throw in a curve ball where he confuses passion and competence, and I did take him to task on that one at the time, but that aside, the whole point he was making was actually a valid one. What evidence have you made publicly available that you really do have the skills listed on your CV or your LinkedIn profile?

Sadly, for well over 90% of developers out there in the market for a job, the answer is: none whatsoever.

I’ve heard plenty reasons why this might be the case, but I’m yet to hear a good reason for it. For example, some people claim that requiring a GitHub account discriminates against busy people and people with families. That seems like just making excuses to me. You don’t have to spend five hours a day at it, or even five hours a week — and a recruiter expecting something of that order probably would be discriminating unfairly. But if you can’t manage to rustle up five hours in a year to put together one or two small but well-written projects to showcase when you’re looking for a job, do you really have your priorities right?


Cueball: There’s a reason for everything.
Megan: Yeah, but it’s not always a good reason.
—”Time”, xkcd

This certainly wouldn’t be accepted in certain other lines of work. You wouldn’t hire a photographer or an architect who didn’t have a portfolio, for example, nor would you take an academic seriously without the all-important track record of peer-reviewed publications in scholarly journals. With that in mind, it seems a bit odd to me that we shouldn’t have something similar in the very industry that practically invented the online portfolio. Or that having something similar, we fail so spectacularly at actually making use of it.

The gold standard here is, of course, an active GitHub account. Some people have objected to the concept of GitHub as your CV in recent months for various reasons, but none of them have come up with any better suggestions, and the fact remains that even just one or two public GitHub or Bitbucket pull requests or similar shows that your code has been reviewed and endorsed by other developers. But even if you haven’t had any pull requests, any code up there is better than nothing. Even simple snippets will do—GitHub lets you post gists on https://gist.github.com/ just by copying and pasting from your IDE to your browser. In the absence of code itself, informed discussions about programming still carry some weight. A blog, or some answers on Stack Overflow, are both ways of supporting your credentials.

In the end of the day, being a dark matter developer probably won’t stop you getting a job, and similarly an online presence won’t guarantee you one. It simply isn’t feasible at this stage to systematically reject everyone who isn’t on GitHub, as some people advocate — especially in the .net ecosystem which is still frustratingly conservative and traditional. But having some code publicly available for hiring managers to review could certainly make the difference between your CV being noticed and it being lost in a pile along with thirty other CVs that all look exactly the same as each other. Besides, things are changing, and I sometimes wonder — five years or ten years from now, might dark matter developers end up finding themselves unemployable?

10
Dec

Mercurial is doing better than you think

Here’s something interesting that I came across the other day. Apparently, Facebook has recently switched from Git to Mercurial for source control for its internal projects, and on top of that, they have hired several key members of the core Mercurial crew, including Matt Mackall, the Mercurial project lead.

Their main reason for this decision is performance: some of Facebook’s repositories are so large that they are bringing Git to its knees. They looked at the Git source code and the Mercurial source code and felt that the latter would be easier to fine tune to give them the performance that they needed.

This is an interesting development. With Git now on the verge of overtaking Subversion to the most widely used SCM in corporate settings, it’s tempting to write off Mercurial as something of a lost cause. Back in January, when I posted a suggestion on the Visual Studio UserVoice forums that Microsoft should support Mercurial as well as Git in Team Foundation Server, I thought it would be doing well to get three hundred votes and then plateau. But as it stands, it’s now passed 2,500 votes and still going strong, making it the tenth most popular open request on the forums and the second most popular request for TFS in particular, with nearly twice as many votes as the original DVCS request had. Git may have cornered the market for open source collaboration, but its unnecessarily steep learning curve and often pathological behaviour make it surprisingly unpopular with the majority of developers for whom public collaboration on open source projects is not a priority.

It’ll be interesting to see the outcome of this, but one thing is for certain: it’s not all over yet.

04
Nov

So I’ve built my own ALT.MVC engine

(ALT.MVC means any MVC-style framework for .NET that was not written by Microsoft.)

For the past few weeks, my commute-time hobby project has headed in a completely new direction, and I’ve been working on a completely new MVC framework for .NET. I was inspired to do this mainly by some of the other offerings in the ALT.MVC space, such as NancyFX, FubuMVC, Simple.Web and OpenRasta, and I was curious to see just how much effort is involved in getting a minimum viable framework up and running. About two weeks, it turns out, plus about the same again to add some spit and polish, though the end result is pretty spartan. It doesn’t (yet) have any kind of model binding, or validation, or CSRF protection, or authentication and authorisation, or even session management. But these will come in due course.

Different MVC frameworks have different emphases. ASP.NET MVC is generally designed to appeal to as broad a constituency as possible. As a result it’s a bit of a jack-of-all-trades: it does everything under the sun, most of it passably, some of it awkwardly, and none of it brilliantly. FubuMVC is designed for people who are steeped in Clean Code and design patterns, and who have a degree in Martin Fowler. NancyFX is built around the “super-duper-happy path”: its aim is to keep things as low-ceremony and low-friction as possible. WebForms was designed to ease the transition to web development for traditional VB and Delphi developers, who are used to event-driven drag-and-drop RAD programming. The result is a complete untestable mess, simply because the web doesn’t work that way.

The key emphasis behind my own MVC effort is modularity. Most MVC frameworks encourage a fairly monolithic design to your application. Sure, they may be extensible using IOC containers and so on, but you’ll have to jump through several hoops in order to implement feature toggles, or A/B tests, or plugin frameworks, and even then your solution will be partial at best. My MVC framework, on the other hand, is built entirely around the concept of pluggable, switchable modules. This allows for some unique features. For example, you can have your static files—your CSS, JavaScript and images—switchable along with their corresponding server-side components. Turn off the feature that serves up /my/secret/script.js and it will give you a 404 when you try and access it in your web browser.

Anyway, enough of the sales pitch, let’s take a look at how it all works. A module looks like this:

public class HomeModule : Dolstagis.Web.Module
{
    public HomeModule()
    {
        AddStaticFiles("~/content");
        AddViews("~/views");
        AddHandler<Index>();
    }
}

This tells us that all files within your content folder should be served up as static files; that your views are all in your views folder, and registers a handler called Index to handle your home page. Handlers might look something like this:

[Dolstagis.Web.Route("/")]
public class Index : Dolstagis.Web.Handler
{
    public object Get()
    {
        var model = new { Message = "Hello world" };
        return View("~/views/hello.mustache", model);
    }
}

This class handles all GET requests for the home page with the view at views/hello.mustache using an anonymous class for our model. I’m using Nustache as the default view engine at present; since these are the same as Mustache views, you can easily use them on both client and server. Note that you can only have one route per handler: this is by design as it is much closer to the Single Responsibility Principle, and reduces the number of dependencies that you have to inject into your controllers that end up not even being used on most of your requests.

As I say, it’s still very much early days yet, and it’s not production ready by a long shot, but there are several things I’ve got planned for it. If you fancy getting involved with something new in the ALT.MVC space, then I’m always open to suggestions and pull requests. You can find the source code in my Dolstagis.Web repository on GitHub, or drop me a line in the comments below or to @jammycakes on Twitter.

07
Oct

Your best practices are (probably) nothing of the sort

Now I have nothing against best practices per se.

But if you are going to tell me that something is a “best practice,” please first make sure that it really is a best practice. The software development world is plagued by so-called “best practices” that are nothing of the sort, that just introduce friction, ceremony and even risk without offering any benefits whatsoever in return. Some of them were once perfectly valid but have been superseded by developments in technology; some of them were based on widely held assumptions that have since been proven to be incorrect; some of them are based on total misunderstandings of something that someone famous once said; and some of them are just spurious.

I’ll give one example here, which came up in a discussion on Twitter the other day. It’s quite common for people to put their interfaces in one assembly, their business logic in a second, their repositories in a third, their models in a fourth, their front end in a fifth, and so on. This is all done in the name of “having a layered architecture.” The problem with this is that it makes dependency management harder (in fact in the pre-NuGet days it was an absolute nightmare) and forces you to jump around all over the place in your solution when making changes to related classes. It just adds friction, without even solving the problem it claims to solve: separate assemblies are neither necessary nor sufficient for a layered architecture. Oh, and it also violates the Common Closure Principle, which states that classes that change together must be packaged together.

Unfortunately, these so-called “best practices” proliferate because most developers lack the courage to question them, for fear of being viewed as incompetent or inexperienced by those with the authority to hire, fire or promote them. The people who promote garbage “best practices” tend to have Many Years Of Experience At Very Impressive Sounding Companies, and if you’re not that experienced (or confident) yourself, that can be quite intimidating. You don’t agree that we should put our interfaces, enums, business classes, repositories and presentation layers in separate assemblies? You obviously don’t understand a layered architecture!

Don’t let that intimidate you though. When somebody tells you that “you’re not following best practices,” it’s an indication that in their case, Many Years Of Experience At Very Impressive Sounding Companies actually means one year of experience repeated many times building run of the mill CRUD applications on outdated technologies at places that store users’ passwords in plain text. They are almost certainly not active on GitHub, or Twitter, or Stack Overflow, they are very unlikely to have hobby projects, and they probably never discuss programming with experts from outside their own team, let alone from other technology stacks.

In other words, The Emperor Has No Clothes.

But when something really is a best practice, it’ll be quite different. For starters, they will cite the practice concerned by name. They won’t tell you that “you’re not following best practices” but that “you’re violating the Single Responsibility Principle” or “you’re making test driven development harder” or “You Ain’t Gonna Need It” or something else specific. Another hallmark of a genuine best practice is that it will have tangible, enumerable benefits that are actually relevant to your situation. Here are some questions you can and should ask about it:

  1. Does it make it easier to get things right?
  2. Does it make it harder to get things wrong?
  3. Does it make it easier to back out when things go wrong?
  4. Does it make it easier to diagnose problems?
  5. Does it make it easier to get things done faster and with less effort without compromising points 1-4?
  6. Does it deliver the benefits that it claims to deliver? What evidence do you have that it does?
  7. Does it solve problems that you are actually likely to face, or is it one big YAGNI-fest?
  8. Are the problems that it solves still relevant, taking into account the current state of technology, market forces, security threats, and legislative or regulatory requirements?
  9. What alternative approaches have you considered, and how did they compare? It’s nonsensical to talk about “best practices” when you have nothing to compare them against, because the word “best” is meaningless in a sample size of one.
  10. Do its benefits actually outweigh its costs? In practice? In your situation?
  11. Have you understood it correctly, and are you sure you’re not confusing it with something else?

Any best practice that is worth following will stand up to scrutiny. And scrutinised it should be. Because blindly doing something just because somebody cries “best practices” is just cargo cult. And cargo cult programming is never a best practice.

09
Sep

My choice of Git GUI tools

Despite Git’s reputation for being ridiculously command-line centric, Git users are actually pretty spoilt for choice when it comes to graphical front ends. Here’s what I use.

Atlassian SourceTree has been around for a while on the Mac, but it has only recently seen a 1.0 release for Windows. It is free, though it does require user registration.

SourceTree screenshot

In terms of features, SourceTree does just about everything I want it to, and visually it’s the one I find easiest on the eye. One particularly nice feature of SourceTree is that it automatically fetches from all your remotes every ten minutes, so you’re quickly kept abreast of what your colleagues are working on. My main gripe with it is that “Push” to a Subversion repository doesn’t work, even though it does bring up a dialog box saying that it will, so I have to drop to a command prompt to type git svn dcommit. I’d also like to see an interface to git bisect, though no doubt that will come in due course.

For integration with Windows Explorer and Visual Studio, I use Git Extensions:

Git Extensions screenshot

I don’t use it for that much else, mainly because I find it visually a bit harsh, but it’s pretty fully featured. One nice feature is the ability to recover lost commits — useful if you do an unintended git reset --hard.

If you find that Git Extensions doesn’t show up in Visual Studio 2012, you may need to explicitly tell Visual Studio where to find it. This thread on the Git Extensions Google group tells you what you need to know.

My merge tool of choice is Perforce Merge:

p4merge

It’s a really nice tool that’s easy on the eye, easy to use, and gives you a very clear view of what’s changed. If you’re still using TortoiseMerge or kdiff3, you’re missing out.

06
Aug

Namespaces in JavaScript

Namespaces are a common technique used in many programming languages to avoid naming collisions between different parts of your code. Unfortunately JavaScript doesn’t have built in namespace support, but it can be implemented fairly simply by creating a nested hierarchy of objects in the global namespace. A bit like this:

var My = My || {};
My.Cool = My.Cool || {};
My.Cool.Namespace = My.Cool.Namespace || {};

My.Cool.Namespace.showMessage = function(msg) {
    $('#messageBox').html(msg).show();
};

The problem with this is that it’s pretty verbose. It would be far better if you could have a function to declare a namespace and simplify things.

There are several different approaches to this knocking around. Usually, they look something like this:

var ns = namespace("My.Cool.Namespace");
ns.showMessage = function(msg) {
    $('#messageBox').html(msg).show();
};

My approach is a little bit different. Your namespace function takes a second argument, a function which is called to initialise your namespace. You can do this by assigning properties to the this object in the function body:

namespace("My.Namespace", function() {
    "use strict";

    this.showMessage = function(msg) {
        $('#message').html(msg).show();
    };
});

My.Namespace.showMessage('Hello world');

The advantage of this approach is that it implements the JavaScript module pattern, enclosing the entire contents of your file in a single function. Another thing I’ve implemented is the ability to create shortcuts to other namespaces. You can do this by passing them in an additional array to the namespace() function, which then forwards them on to the initialiser:

namespace("My.Namespace", [jQuery, Backbone], function($, bb) {
    "use strict";

    // bb is a shortcut for the Backbone namespace of
    // backbone.js; this example creates a Backbone model
    // called My.Namespace.Note

    this.Note = bb.Model.extend({
        initialize: function() { ... },

        author: function() { ... },

        coordinates: function() { ... },

        allowedToEdit: function(account) {
            return true;
        }
    });
});

I’ve put my namespace function on GitHub, along with some more detailed instructions on how to use it.

13
Jun

The writing is on the wall for Subversion as Git takes over

This time last year, the Eclipse Community Survey noted that Git’s market share had risen from 12.8% to 27.6%, while Subversion had dropped from a seemingly unassailable 51.3% to 46.0%. This year’s survey results, published yesterday, note that this trend has continued: Git/GitHub has risen to 36.3% while Subversion has dropped to 37.8%. Subversion may still be in the top slot for now, but its lead is tiny and it is rapidly losing ground.

Other data sources, such as itjobswatch.co.uk, paint a similar picture. Look at how demand for Git skills has grown in recent years:

Git demand according to itjobswatch.co.uk

Job trackers such as this tend to give Subversion a bigger lead, because they focus on the rather more conservative corporate market and purposely ignore the world of hobbyists and open source developers. But even so, the trend is clear. Thirteen percent of UK programming jobs now ask for Git experience. Seventeen percent ask for Subversion, but the gap is narrowing rapidly and it is almost certain now that Git will overtake Subversion in corporate settings by the end of this year.

We are now fast approaching the point at which not using Git will increasingly hurt developers and companies alike. As a developer, a lack of Git experience is now starting to call into question your willingness and ability to keep your skills up to date. As a company, if you don’t use Git, you will find yourself competing for good developers against companies who do. Once you’ve got used to Git, Subversion is a painful experience, and fewer and fewer competent developers will be prepared to put up with it given the choice.

Then there are third party products and services. Already we are seeing an increasing number of these coming on the market which only support Git — GitHub and Heroku being two prominent examples. Those that do support other alternatives are increasingly treating them as an afterthought, with only limited features. Even if you’re a Microsoft-only shop, Git is getting harder to avoid. Entity Framework and ASP.NET MVC, along with several other Microsoft-run projects, are now hosted using Git. Team Foundation Server is introducing Git as a first-class source control option, complete with the tight end to end integration experience which TFS users value so much. Windows Azure makes Git one of its main avenues for deployment.

Not only has Subversion fallen behind, its development is painfully slow. Subversion 1.7, originally scheduled for the spring of 2010, was only released in October 2011 — a year and a half late. Subversion 1.8 is also a year late and has had its scope cut back by a half. Subversion 1.9, tentatively slated for this time next year, could well see even more significant delays, especially if the shift in demand forces its key players to divert resources to Git-based products and services. Subversion 1.10, the first to promise some genuinely useful new features (shelving and checkpointing), is “speculatively at best” scheduled for mid-2015. It is quite possible that it may never be released.

Subversion has no future. It is old, obsolete, decrepit technology and you need to be planning for its end of life. Git, on the other hand, is rapidly becoming the lingua franca of source control throughout the entire software industry. Love it or hate it, but if you don’t take it seriously, it won’t be long before the industry doesn’t take you seriously.

30
May

Dolstagis: my pet project

I thought it would be a good idea to say a bit about my pet project that I’ve been working on over the past few months on my daily commute.

Round about September or October, I ended up reading Patrick McKenzie’s blog, kalzumeus.com, where he was talking about how he managed to start up a business from his hobby programming project, Bingo Card Creator, on no more than five hours a week. Seeing as I spend twice that sitting in trains, otherwise doing nothing other than playing Angry Birds, staring out of the window, reading xkcd, or sleeping, I thought that something like that would be a much more profitable use of my time.

Only one problem: I didn’t have any ideas. Or rather, I had too many of them. At any one time I will have half a dozen ideas for a web application floating around in my head, but none of them have yet risen above the others. Nevertheless, I figured that there’s a lot of ground work to be getting on with before you get your big idea, so I typed hg init, cranked up Visual Studio, and got going. I’d also heard the story of how Flickr got started: how it was originally intended to be an online game but ended up as a photo sharing service almost by accident, and I figured that perhaps one particular idea would emerge out of the melting pot as I get working on it.

So far I still haven’t had my big idea, but I have ended up with the makings of a web application framework for ASP.NET MVC. Its intentions are similar to those of Django, the Python web application framework, in that it’s a “batteries included” framework which will eventually offer all sorts of building blocks for your own ASP.NET applications: dependency injection, unit and integration testing, NHibernate session management, asset bundling and minification, user authentication and authorisation, an admin section, comments, and eventually even integrating unit testing for JavaScript into your build process.

It’s also turned out to be a bit of a playground for me to experiment with new things, and to try out some of my ideas and hypotheses about patterns and practices to see if they’re any good. So far, I’ve tried (and rejected, for now at least) CoffeeScript, the Web API, and not using the Repository pattern.

Since I suck at coming up with cool names for things, I’ve called it “Dolstagis” for now, after an in-joke that was current among some of my colleagues and myself at work a few years back. I’ll no doubt write a bit more about it over the coming weeks and months, and the lessons that I’ve learned along the way, but in the meantime, if you want to see what it looks like, I’ve posted the code on GitHub.