james mckay dot net
because there are few things that are less logical than business logic

Author Archive

Some thoughts on DevOps

It’s now six weeks since I started at my new job, and I’m really enjoying it. Returning to .NET has felt like a homecoming in many ways. Even though I’ve been quite critical of some of the things that go on in the Microsoft ecosystem at times, it’s what has paid the bills for most of the past sixteen years, it’s a platform that I enjoy working with, and I’d built up quite a lot of experience and expertise in it in that time.

My two year hiatus from .NET was spent mostly in the world of DevOps and cloud computing with AWS. While I gained some valuable experience with it, I never really settled down in it. In particular, I was especially unhappy about being pigeonholed as a “WebOps engineer” on what our delivery manager insisted was “an Ops team.” I’m a developer, not an Ops guy, and besides, that kind of thinking completely flies in the face of what DevOps is supposed to be all about.

If you’re calling your team an “Ops team,” you’re not doing DevOps.

There’s a very good reason why DevOps is called DevOps and not OpsDev or Ops. It is Development first and Ops second. Or, if you want to put it a different way, it is about the Development of a software product to automate your Ops. Jez Humble, who wrote the book on Continuous Delivery, tells us that there’s no such thing as a DevOps team for a good reason. In the DevOps world, Ops is a software product, not a team.

This being the case, while you may need experienced Ops specialists to give you direction on what needs to be built, you also need experienced developers to build it. They need to have a thorough grounding in concepts such as design patterns, the SOLID principles, dependency injection, separation of concerns, test-driven development, algorithmic complexity, refactoring, and the like. You need to recruit, promote, plan, prioritise, and provide training accordingly. Otherwise you’ll either limit what you’re able to achieve, or else you’ll end up with unmaintainable code that needs to be rewritten. And when you’re dealing with infrastructure as code, a rewrite is far, far harder than when you’re dealing with business logic.

In any case, DevOps needs to be the responsibility of your development team as a whole. The whole point of DevOps is to break down the silos between Development and Ops, and to have a separate DevOps team (or worse, a separate Ops team) just creates another silo that you could be doing without.

Your Repository is not a Data Access Layer

The Repository pattern has come in for a lot of criticism over the past few years by high-end .NET developers. This is understandable, because in most projects, the Repository layer is usually one of the worst-implemented parts of the codebase.

Now I’ve been critical of badly implemented Repositories myself, but to be fair, I don’t think we should ditch the pattern altogether. On the contrary, I think that we could make much more effective use of the Repository pattern if we just abandoned one popular misconception about it.

Your Repository is (mostly) not a DAL.

If you’re wondering what I mean, here is an example of a typical Repository method. It comes from BlogEngine.net, an open source ASP.NET blogging platform, and it is typical of the kinds of Repository methods that you and I have been working with on a daily basis for years:

public CommentsVM Get()
{
    if (!Security.IsAuthorizedTo(Rights.ViewPublicComments))
        throw new UnauthorizedAccessException();

    var vm = new CommentsVM();
    var comments = new List<Comment>();
    var items = new List<CommentItem>();

    var all = Security.IsAuthorizedTo(Rights.EditOtherUsersPosts);
    foreach (var p in Post.Posts)
    {
        if (all || p.Author.ToLower() == Security.CurrentUser.Identity.Name.ToLower())
        {
            comments.AddRange(p.Comments);
        }
    }  
    foreach (var c in comments)
    {
        items.Add(Json.GetComment(c, comments));               
    }
    vm.Items = items;

    vm.Detail = new CommentDetail();
    vm.SelectedItem = new CommentItem();

    return vm;
}

Now this isn’t bad code. It’s actually quite clean code. It’s clear, well-formatted, and easy to understand, even if returning a ViewModel from your Repository does make me twitch a bit. But where is the data access logic?

There is not a single line in this code that tells me which underlying persistence mechanism is being used. Are we talking to Entity Framework? To NHibernate? To RavenDB? To a web service? To Amazon DynamoDB? Or to a program for comparing human and chimp genomes? In just about every .NET project that I’ve encountered, the Repository classes are all populated with methods just like this one. They may contain some LINQ queries, but these won’t give me any indication either. Yet in every single case, they’ve been in projects called My.Project.DAL or something along those lines.

We’re sometimes told that the role of the Repository layer is to abstract away your data access logic from your business logic. But in methods such as this, the data access logic appears pretty thoroughly abstracted to me already.

No, this is business logic, pure and simple.

Why we’ve been thinking of the Repository as a DAL

The reasons why Repositories are viewed as a data access layer are purely historical. The classic three-layer architecture dates back to the late 1990s, when everybody thought that stored procedures were the One True Best Practice™, and that moving your BLL and DAL onto separate hardware was the right approach to scalability problems that almost nobody ever had to face in the Real World. Back in the early days of .NET 1.0, your typical Repository contained method after method that looked something like this:

public User GetUser(int userID)
{
    using (SqlConnection cn = new SqlConnection(connectionString))
    using (SqlCommand cmd = new SqlCommand("usp_GetUser", cn)) {
        cmd.Parameters.Add(new SqlParameter("@UserID", userID));
        cmd.CommandType = CommandType.StoredProcedure;
        using (SqlDataReader reader = cmd.ExecuteReader()) {
            if (reader.Read()) {
                return new User(
                    (int)reader["UserID"],
                    (string)reader["UserName"],
                    (string)reader["DisplayName"],
                    (string)reader["Email"]
                );
            }
            else {
                return null;
            }
        }
    }
}

It was pretty much in-your-face that this was data access code. It was also very, very tedious and repetitive to maintain. It was this tedium that gave rise to modern O/R mappers, and in fact, in the early days, offerings such as LLBLGen Pro and NHibernate were sometimes actually referred to as “generic DALs.” Then, eventually, Microsoft got in on the act with Entity Framework.

In a nutshell, your data access layer is now Entity Framework itself.

Your Repository is first and foremost business logic

The problem with viewing modern-day Repositories as a DAL is that it demands that you draw a clear distinction between data access logic and business logic while obfuscatinig that very distinction.

I’m yet to see a clear, coherent definition of where the distinction lies. The nearest I can get is a vague and woolly concept of LINQ code as being data access on the grounds of equally vague and woolly concepts of IQueryable<T> being tight coupling. Now Mark Seemann makes some valid points in his blog post — LINQ is indeed a leaky abstraction — but what that means in practice is that if you run up against the leaks in the abstraction, you are dealing with inseparable concerns, which simply can’t be categorised cleanly as either business logic or data access logic, and have to be tested using integration tests rather than unit tests. Another example of inseparable concerns is where you have to bypass Entity Framework altogether to go directly to the database, for example, for performance reasons.

In fact, LINQ may be a leaky abstraction, but it’s a much better abstraction than any alternative you’re going to come up with. Once again, LINQ code gives you no indication whatsoever of what underlying data access mechanism you are actually using, and in many cases you can — and should — test anything you do with IQueryable<T> without hitting the database. In any case, query construction implements business rules and is therefore well and truly a business concern.

So what is the Repository pattern, as implemented in most projects, best for? Simple: a query layer. While query objects are a better choice for more complex queries, and extension methods on IQueryable<T> should be considered seriously for cross-cutting concerns such as paging and sorting, for simpler queries with only a few arguments each, a Repository is not a bad choice.

The Great Shorts Debate

It’s the height of summer. The Great British Heatwave is in full swing. People are questioning whether their office dress codes are fit for purpose. And this means one thing in particular: The Great Shorts Debate.

Now I work for a company where I am allowed to wear shorts to work whenever I like without anyone so much as batting an eyelid. However, not everybody is so fortunate. Far too many workplaces are still saddled with stuffy medieval dress codes that demand that office workers turn up to work in winter woollies long trousers even in a heatwave. It seems that every year we see news reports of men and boys wearing skirts in protest against such bureaucratic and pointy-haired nonsense.

Regardless of what you think of such protests, there is no valid reason whatsoever why office workers should not be allowed to wear shorts to work.

I’ll just say it: corporate dress codes that do not allow men to wear shorts to work are discrimination, it’s as simple as that. Long trousers are sweaty, uncomfortable, restrictive and stifling in warm weather. They make you feel grumpy and irritable, while increasing the likelihood that you’ll spend the last hour of the day watching the clock for the minute you can get out the door and change into something more sensible. They lower productivity while contributing nothing whatsoever to the bottom line.

Nobody’s asking you to adopt an “anything goes” dress code here, with ripped cut-off jeans, tank tops, socks and sandals, or bare feet in client meetings. Even if you think that cargo shorts are too casual, you can still look crisp and clean in a combination of chino or tailored shorts with a polo shirt or a button-down short-sleeved shirt, ankle socks, and tennis shoes. As for places that require jackets and ties, they figured that one out in Bermuda a century ago. Besides, when even the BBC allows its weathermen to appear on national TV in shorts, what makes you think you have an excuse?

The only legitimate reason why shorts should be off-limits in hot weather is health and safety. If you’re working with dangerous chemicals in a laboratory, for example, you may need that extra protection. But in an office, you aren’t working with dangerous chemicals, so these constraints do not apply. By all means insist that your staff are clean, tidy, professional and sharp in their appearance. But insisting that they turn up to work dressed for winter in the middle of a heatwave is just silly.

To throw or not to throw? Non-exceptional error conditions

I’ve said before that the general advice that “you should only throw exceptions in response to exceptional conditions” is bad advice, because it is ambiguous and unclear as to what exactly constitutes an exceptional condition. But nevertheless, there is a valid point behind it.

The point is that exceptions are (a) slow, and (b) painful to step through in the debugger.

This means that if there are error conditions that crop up frequently, you will want to handle them without an exception being thrown wherever possible.

The classic example here is, of course, attempting to parse an integer. This is why .NET gives you the Int32.TryParse method. If it is provided with an invalid input string, it returns false instead of throwing an exception.

Now there are two important points to note about Int32.TryParse.

First: it returns false only in response to a single, specific error condition: namely, bad user input. Any other failure mode should still throw an exception. This is the case with the overload of Int32.TryParse that also takes some localisation and formatting parameters, for example.

Second: its name tells us that it returns an error code in response to this specific condition. If it didn’t have that Try... prefix, it would, as Yoda said, be a case of “Do or do not. There is no try.” This means that the meaning of exceptions — that the method can not do what its name says that it does — is preserved.

Third: it is provided alongside an equivalent method, Int32.Parse, that throws an exception as normal. This gives consumers of your API a choice about which method to use.

Fourth, and most importantly, it is not implemented by throwing and re-catching exceptions. If you write a method like this, you are doing it wrong:

public bool TryParse(string input, out int output)
{
    try {
        output = Int32.Parse(input);
        return true;
    }
    catch {
        return false;
    }
}

Rather, you should do it the other way round:

public int Parse(string input)
{
    if (input == null) {
        throw new ArgumentNullException("input");
    }
    int result;
    if (Int32.TryParse(input, out result)) {
        return result;
    }
    else {
        throw new FormatException("The parameter 'input' was not in the correct format.");
    }
}

Why should this be? Remember that exceptions are expensive both in terms of execution time and stepping through them in the debugger. If you are implementing Try... methods by wrapping other methods in a try ... catch block, you will be losing the benefits of having a Try... method in the first place.

Recommendation: Use this pattern whenever you are taking user input that may or may not be valid. But do not use this pattern for error conditions that are not a direct result of invalid user input.

Returning.To<.NET>()

Over the past fifteen years or so, it’s been trendy for .NET developers to write blog posts announcing that they’re “leaving .NET”. In this blog post, I am going to do the exact opposite. After a two year hiatus, I will soon be working once more with Microsoft technologies.

At the end of May, I will be leaving the Parliamentary Digital Service.

I’ve been working for the Houses of Parliament for nine and a half years now, and it’s been an awesome and exciting privilege. I’ve had a ringside seat at three General Elections and the Brexit referendum. I’ve not only seen the political process close up, I’ve had to translate it into computer code. I’ve learned things about the ins and outs of the running of the country that I never even knew existed. It’s felt like Alice in Wonderland, but it’s been an amazing time.

I will be having a week’s holiday and then starting a new job at JFA Systems, a small company based in Horley that specialises in creating software for the travel industry, on the fourth of June.

What kind of .NET shop?

Some people may find it a bit curious that I’m returning to .NET, given just how scathing I’ve been in the past about some of the things that you see in the .NET ecosystem.

But not all .NET shops are places that insist on being spoon-fed by Microsoft and don’t let you think for yourself. Not all .NET shops still think that stored procedures are a Best Practice and refuse to move forward. Not all .NET shops demand that you rewrite perfectly good Python build scripts in unmaintainable MSBuild gibberish for no reason whatsoever other than That Is How Microsoft Wants You To Do It. Not all .NET shops are hostile to open source or interacting with the wider software development community.

Certainly, I left the interview at JFA Systems with the distinct impression that they were on the same page as me about this. I was every bit as scathing about the SFBMs and cargo cultists in my interview as I have been on my blog — and they loved it. When they offered me a job later in the day, I had no hesitation in accepting it.

One thing in particular that I’m looking forward to is getting back into full-stack development. For most of my career I’ve worked with a healthy mixture of front-end, back-end and API development as well as Ops/DevOps/deployment. In fact, the original intention two years ago, when we first started prototyping a continuous delivery pipeline on AWS for the new Parliament website, was that I should get some experience with modern DevOps techniques and cloud platforms to augment my skills and experience as a full-stack developer. But things changed, and the team I was on ended up becoming first and foremost an Ops team. They’ve valued my contributions as a developer, and as such making it more DevOps than just plain Ops, but I’ve never been entirely happy about becoming a DevOps specialist.

So long, and thanks for all the jerk chicken…

If there’s one thing that that I’ll miss about working in Parliament more than anything else, it’s the House of Commons catering department’s most popular dish — jerk chicken with rice and peas.

Journalists and political pundits sometimes complain about the fact that food in Parliament is subsidised, so that over-paid and over-expensed MPs can eat luxurious mouth-watering food on the cheap while the hoi polloi outside the Westminster bubble has to pay through the nose for stuff that tastes like cardboard. To be honest, I don’t think such criticisms are all that fair. Food in Parliament is subsidised for everyone on the Parliamentary estate and their guests — including the journalists making such criticisms in the first place. Besides, at some tech companies and startups — most famously Google, for example — the food isn’t just subsidised, it’s free.

Fortunately, for jerk chicken lovers who do not enjoy access to the House of Commons, all is not lost. Its recipe is open source.

Trains! Trains! Trains!

I may miss the jerk chicken, but there’s one thing that I will be glad to see the back of, and that’s taking the train into London. For the past nine and a half years, I have been getting up far earlier than I’m comfortable with, spending between three and a half and four hours a day commuting, and paying up to £4,000 a year to Southern Railway for the privilege. It’s a gruelling, exhausting, stressful, expensive and wasteful regime and I wouldn’t recommend it to anybody.

By contrast, the commute to Horley from where I live is a pleasant half-hour drive, half of it through pretty Sussex villages, and half of it up the A23/M23 from Handcross to Gatwick. I love driving, and I actually find it helps me to unwind. I’m looking forward to being able to set my alarm for 7am, leave home just before 8:30, and then get back home shortly after 6pm.

It will also be significantly cheaper. I estimated that it will probably cost me around £1,000 a year in petrol to get to and from Horley. An annual season train ticket, on the other hand, costs over £1,600, while the train journey itself would take significantly longer and put annoying constraints on when I could travel.

New adventures

Change can be daunting and unnerving, and perhaps even scary, and it can also take a lot of effort. But once you step out, it can actually re-ignite passion and enthusiasm into your career. I’m looking forward to new horizons and new opportunities, and I’m once again excited about what the future holds. My last day in Parliament is 25 May, after which I will be taking a week’s break before starting at JFA on the fourth of June.

I’d particularly like to thank Paul Wilson of Arc IT Recruitment for linking me up with JFA Systems. Recruiters often get a bad reputation among developers and employers alike, but I’ve been thoroughly impressed with the professionalism and integrity that I’ve found in Paul and his team.

Finally, I’d like to thank my colleagues both past and present in the Parliamentary Digital Service for putting up with me over the past nine and a half years. They’re doing a great job playing their part in keeping the country running smoothly, and I wish them all the best for the future.

The Palace of Westminster by night, December 2008. Photo by me.