Martin Fowler and feature branches revisited

Fábio asks me this by e-mail:

I found this very interesting article ( that once was in your blog, and it seems it can only be accessed in the archive, or at least I was not able to find it in the live version of your blog.

Is there a reason for that? Do you still keep the same opinion? If not, can you give me a quick hint of what to read to reach the same conclusion?

In answer to his question, I took the original blog post down in January 2013 along with everything else on my blog, because I wanted to give it a complete reboot. In the years since then, I’ve restored some of them going as far back as 2009,  but I hadn’t restored that particular post, mainly because I felt that I’d been too confrontational with it and it hadn’t made me look good. However, since it’s out there in, and people are still asking about it, I’ve now restored it for historical reference. Unfortunately, I haven’t restored the comments because I no longer have a backup of them.

A bit of historical background.

I wrote the original post in July 2011. At the time, distributed version control tools such as Git and Mercurial had been around for about six years, but were still very new and unfamiliar to most developers. Most enterprise organisations were deeply entrenched in old-school tools such as Subversion and Team Foundation Server, which made branching and merging much harder and much more confusing than necessary, most corporate developers were terrified of the concept, and vendors of these old-school tools were playing on this fear for all it was worth. This was intensely frustrating to those of us who had actually used Git and Mercurial, could clearly see the benefits, and yet were being stonewalled by 5:01 dark matter colleagues and managers who didn’t want to know. To us, Subversion was The Enemy, and to see someone of Martin Fowler’s stature apparently siding with the enemy was maddening.

On the other hand, these tools weren’t yet quite enterprise ready, with only weak Windows support and rudimentary GUIs. SourceTree was not yet a thing. Furthermore, the best practices surrounding them were still being thrashed out and debated by early adopters in the blogosphere and at conferences. In a sense, nobody properly understood feature branches yet. If you read all the discussions and debates at the time, we didn’t even agree about exactly what feature branches were. Martin Fowler, Jez Humble and others were working with the assumption that they referred to branches lasting several days or even weeks, while people such as Adam Dymitruk and myself went by a much broader definition that basically amounted to what we would call a pull request today, albeit with the proviso that they should be kept as short as possible.

These days, of course, everybody (or at least, everybody who’s worth working for) uses Git, so that particular question is moot, and we can now freely discuss best practices around branching and merging as mature adults.

The state of the question in 2017.

I’m generally somewhat more sympathetic to Martin Fowler’s position now than I was six years ago. Most of his concerns about feature branches are valid ones. Long-lived feature branches can be difficult to work with, especially for inexperienced teams and badly architected codebases, where big bang merges can be a significant problem. Feature branches can also be problematic for Continuous Integration and opportunistic refactoring, so they should very much be the exception rather than the rule. Feature toggles can also provide considerable benefits. You can use them for A/B testing, country-specific features, premium features for paying customers, and so on and so forth. I even started writing my own .NET-based web framework built around feature toggles, though as I’m no longer working with .NET, it’s not being actively developed.

However, many of my original points still stand. Short-lived branches, such as pull requests, are fine, and should in fact be the default, because code should be reviewed before it is integrated, not after the fact. Furthermore, if you’re using feature toggles solely as a substitute for branching and merging, you are releasing code into production that you know for a fact to be immature, untested, buggy, unstable and not fit for purpose. Your feature toggles are supposed to isolate this code of course, but there is always a risk that the isolation could be incomplete, or that the toggle could be flipped prematurely by mistake. When feature branches go wrong, they only go wrong in your development environment, and the damage is relatively limited. When feature toggles go wrong, on the other hand, they go wrong in production—sometimes with catastrophic results.

Just how catastrophic? The feature toggles article on Martin Fowler’s own website itself cites an example where badly implemented feature toggles cost one company $460 million in just 45 minutes. The company concerned, we are told, “went from being the largest trader in US equities and a major market maker in the NYSE and NASDAQ to bankrupt.” To be fair, there were other problems—they relied extensively on manual deployment processes, for example—but it is still a cautionary tale for anyone who thinks feature toggles should always be viewed as a Best Practice without exception.

Of course, not all feature toggles carry this level of risk, and in many cases, the effects of inadvertent exposure will be mostly harmless. In these cases, feature toggles will indeed be the better option, not least because they’re easier to work with. But in general, when deciding whether to use a feature branch or a feature toggle, always ask yourself the question: what would be the damage that this feature would cause if it were activated prematurely? If it’s not a risk you’re prepared to take, your code is better off on a separate branch.

Why feature switches?

The key feature of Dolstagis.Web is that it builds the concept of feature switches and Branch by Abstraction right into its core. To anyone who recalls the position that I adopted three years ago in The Great Feature Branch Debate, this will no doubt come as a bit of a surprise. Am I jumping the shark here?

Not really. I may have been objecting at the time to the black-and-white “feature branches bad, feature toggles good” line being adopted by Martin Fowler and his ThoughtWorks colleagues, but my own opinion is somewhat more nuanced than the polar opposite of “feature toggles bad, feature branches good.” There are actually a lot of clever things that you can do with feature toggles:

  • You can set them to flip at a specific date and time, for example, if you want to launch something at a conference.
  • You can have them go by IP address. For example, country-specific features.
  • You can use them for A/B testing.
  • You can use them to turn off features and degrade your site’s performance gracefully under load.
  • You may occasionally have a feature that needs to be turned off in order to perform a specific upgrade.

The problem with feature switches is that when they are used as an alternative to branching and merging, you are regularly deploying code into your production environment that you know for a fact to be buggy, immature and incorrect. If your feature switches aren’t properly architected, this can end up being exposed to the general public — with disastrous results. Breaking your tasks down into small user stories can help, but this is not always the case.

The commonest mistake here is that your feature switch doesn’t switch everything. On an ASP.NET web application, for example, it may switch out your controllers, or your routes, but you will still have the same views, the same JavaScript, the same CSS and the same images and other static assets available for both the “on” and “off” states. It’s difficult to turn static assets on or off when by default they’re all served up by IIS from your web application’s filespace.

Dolstagis.Web gets round this problem quite simply by requiring you to expose your static files and your views explicitly in your feature definition classes. If it hasn’t been added, it won’t be shown: it’s as simple as that. In fact, you aren’t even limited to using your application filespace: you could just as easily include your static files and views as resources within your assembly. You can even have the feature that you are toggling in a separate assembly altogether, and modify your build process so that your production version doesn’t even have a copy of the immature and buggy code.

Of course this doesn’t guarantee you that you’ll get your feature switches right, and there are still ways in which you can get them wrong, but hopefully this approach will help to make it easier to avoid running into problems.

Feature branches versus Continuous Integration

I love asking questions that get people to question their practices and assumptions. Especially when I get the impression that a lot of these practices have been adopted unthinkingly by a lot of people. That is why, in the conversation on Twitter surrounding my critique of Martin Fowler’s position on feature branches, I posed the following two questions:

  1. If DVCS had come first, would Continuous Integration ever have been invented?
  2. What exactly do people understand Continuous Integration to be anyway?

In answer to the second question, you could of course point me to Martin Fowler’s excellent essay on the subject, but I just want to make sure that you’ve read and understood it yourself. For it contains one point that most people seem to overlook when discussing CI, and it is this:

Everyone integrates their work into the main line every day.

For Subversion, this would be trunk. For Git or Mercurial, it would be a single master repository that does not permit multiple heads.

Clearly, most teams do not do this. While there are many tasks that you can break down into functional tasks of less than a day, there are many more that you can’t. Consequently, if you wish to adopt this approach, you will be checking incomplete work potentially into production, hence the need for feature toggles.

It was this, specifically, that I described as a workaround for poor branching and merging support.

Here’s a little thought experiment. Martin Fowler wrote that article in 2006, at a time when few people had even heard of distributed source control. Just suppose that Git had come to prominence ten years earlier and that the DVCS workflows with frequent branching and merging were standard practice industry-wide. Would he have mandated pushing your changes to the mainline on a daily basis then?

Somehow, I don’t think so, and even if he had, it would have gained little or no attention. It would be widely regarded as a solution looking for a problem.

For what it’s worth, long lived branches are a bit of a straw man here. If you are following an agile methodology such as Scrum, your features will naturally be fairly limited in scope anyway so you shouldn’t ever reach that point. Nor am I convinced by the assertion that your mainline should act as a point of communication. That’s what your daily stand-up meetings, sprint retrospectives and face to face communication throughout the day are for.

Of course, there are some situations where a daily integration approach will be beneficial, or even necessary. If you are stuck with an old-school centralised source control system such as Subversion or TFS, for instance, you don’t have much choice. While it is possible in principle to adopt a branch per feature approach with these tools, in practice they enforce so much ceremony and risk around creating, merging, and switching between branches that it simply isn’t practical.

Another scenario is where your team is new to branching and merging. Some novice DVCS users start off with a lot of magical thinking and unrealistic expectations about branching, dive straight in at the deep end with long lived branches and indiscriminate refactoring, then wonder why things go pear shaped when they attempt to merge. My general recommendation is to start with a CI-style approach and gradually expand into small and then larger feature branches as your experience and maturity as a team grows.

Third, some tasks are better suited to feature branches than others. Generally, the more dependencies your code has, the more frequently you will need to integrate. Changes to your data layer or shared classes will need to be integrated pretty much as soon as they’re done, especially if you aren’t following the Open/Closed Principle, whereas on the other hand, changes that largely incorporate new functionality, or that only affect your UI layer, will work fine even on long lived feature branches. Knowing where the limitations and possibilities lie and how to mitigate them allows you to adopt a much more flexible approach to feature branches. But it is certainly possible to adapt your approach to the maturity and expertise of the team, and the needs of your project.

Finally, it is worth noting that feature branches are only incompatible with this one specific aspect of Continuous Integration. Other aspects of CI, and of course pretty much everything about Continuous Delivery, are completely orthogonal concerns, and there is absolutely no reason why you should not do both.

Why does Martin Fowler not understand feature branches?

Update: this subject seems to have generated quite a lively discussion in the blogosphere and on Twitter. If you want to see what people are saying elsewhere and follow the discussion further, here are some links:

Make sure you read the comments on these posts to get both sides of the story.

In the software development world in general, and in the agile community in particular, Martin Fowler is pretty much considered the ultimate authority on just about everything. He’s not like Joel Spolsky, who preaches a feel-good message laced with linkbait such as you should invent your own programming language but not use Ruby, or that the SOLID principles are too bureaucratic to be useful. He speaks in measured tones based on years of experience working with large enterprise clients. He (mostly) shuns controversy and focuses on value. He adopts a Neutral Point Of View. He is not just an agile coach, or an agile luminary, he is one of the authors of the Agile Manifesto itself. He has written several ground-breaking books on software engineering, including Refactoring and Patterns of Enterprise Application Architecture. Someone of that stature makes the best of us look like schoolboys in short trousers by comparison.

So when I came across this video of him and his colleague Mike Mason speaking on the perils of feature branches, I was absolutely horrified. Horrified, because much of what they have to say is totally misleading and completely misrepresents what branch-per-feature is all about. It sounds like the kind of anti-branch FUD that I only ever expected to hear from Highly Paid TFS Consultants and vendors of Subversion-based software, who have the most to lose from the next generation of source control tools that make branching and merging safe, robust and easy.

Note first of all what they’re saying.

  • They are saying that Feature Branching is incompatible with Continuous Integration.
  • They are not saying that you can shoot yourself in the foot if you do Feature Branching wrong, but you can be much more productive if you do Feature Branching right. They are saying that Feature Branching is bad. Period.
  • They are not saying that some tools aren’t cut out for Feature Branching but others are, and if you’re stuck with one of the ones that aren’t, here’s how to make the most of it. They’re saying that Feature Branching is bad. Period.

Now maybe they may argue that they don’t actually mean that, or that I’ve misunderstood them, but the problem is that that is what the anti-DVCS scaremongers will hear, and as sure as eggs are eggs, they will milk it for all it’s worth. Certainly, the whole tone of the video towards feature branches is strongly negative. And in some places, factually inaccurate. Besides which, the alternative that they propose is no better.

Feature branches are not about sporadic integration.

Fowler contrasts feature branching with Continuous Integration as if the two are diametrically opposed to each other. He says this:

The cool rule of thumb that I use for Continuous Integration is that everybody’s integrating back to the mainline at least once a day, while with feature branching, people can go for many days, possibly many weeks, until they complete the feature, and then they only integrate once the feature is done. And it’s that frequency of integration which is the key difference.

No, no, no, no, no, no, no! That is not what feature branches are about!

Feature branches are not about increasing the amount of time between integration, they are about decreasing the time between check-ins. You still break down your features into the smallest tasks you can deliver. You still integrate as frequently as you can, yes, at least once a day where possible. The difference with feature branches is that you start checking in code several times an hour. Why? First, this gives you checkpoints that you can roll back to if you discover that you’ve made a mistake, and second, it makes the thinking behind what you’re doing easier to describe and decipher.

The reason why people think feature branches are for long-running epics is because that is the way that old school centralised tools have taught them to think. They’ve made branching and merging hard through poor tooling, algorithm support and architecture, so people do everything in trunk except for epics, then they run into problems and the whole pattern becomes self-reinforcing. Whereas the correct way to do it is to make branching and merging the default option, so people start off with small, frequent merges, and naturally expand into epics as they develop the maturity to handle them.

Feature branches versus Continuous Integration is a false dichotomy. You don’t do either/or, you do both/and.

Semantic conflicts are not specific to branching and merging.

Fowler then goes on to bring up the issue of semantic conflicts.

And the issue here is, yes they can make the problems of textual merging go away, and certainly the textual merging algorithms have improved greatly over the last ten or twenty years, but there’s still semantic issues that go deeper than the pure text, and they, those do not go away, and as a result you still have big, painful merges and the pain increases exponentially the longer you leave between integrations. So by integrating more frequently, even though you’re doing a lot more integrating, the integrations are sufficiently small that they’re not actually causing a lot of problems and as a result you can do them a lot more smoothly as you go along.

I’m sorry, but that is a straw man. Yes, semantic conflicts are a problem, but they are not specific to branching and merging. Any kind of integration will have semantic conflicts. Trunk-based development will have semantic conflicts. A lock/edit/unlock model such as that adopted by Visual SourceSafe will have semantic conflicts. Everyone editing the same files directly on the same server and not using source control at all will have semantic conflicts. Semantic conflicts are a fact of life no matter what integration model you use. Get used to it.

Besides, the integrations where you need to worry most about semantic conflicts are not the biggest, scariest ones. By the time you get to that level, you are looking at textual conflicts all over the place anyway, so you’ll be on your guard and double checking everything. The semantic conflicts you need to worry about are the ones where there are no textual conflicts at all, where you have dropped your guard. In other words, the size of merges that are typical of Continuous Integration. In any case, we have tools to deal with them. They are called compilers, static analysis, and unit tests. As for any that slip through that net, all you can do is treat them like any other bugs. The question here is, how many of these semantic conflicts translate into new bugs and regressions in your code compared to bugs introduced through any other means? When you are used to feature branching, that is exactly how you view semantic conflicts: as no greater or less a deal than any other form of bug.

For what it’s worth, semantic conflicts have been the subject of some academic research, for example, in this paper, which is well worth a read for anyone who is concerned about them, their prevalence, the risks that they may or may not pose, and strategies to mitigate their effects. But it would be naive and wrong to be paranoid about feature branches because of them, because, as I said, they can crop up just as easily in trunk-based development.

Feature toggles can cause problems too. Bigger, scarier problems.

Feature toggles are a way of avoiding branching and merging, which Joel Spolsky describes as “a workaround for the fact that your version control tool is not doing what it’s meant to do.” Branch by Abstraction is much the same thing, except that it uses IOC containers instead of if statements. Mike Mason explains it as follows:

So, feature toggles are a mechanism whereby you implement a new feature, but simply have it switched off, usually in the software configuration, and in most systems that can be as simple as not including a feature in the UI, so it’s actually being developed under the covers, it’s present in the source code, but you don’t enable it in the UI so nobody actually sees it.

You see what the problem is here? And man, it is a massive problem—visible or not, you are still deploying code into production that you know for a fact to be buggy, untested, incomplete, and quite possibly incompatible with your live data. Your if statements and configuration settings are themselves code which is subject to bugs—and furthermore can only be tested properly in production. They are also a lot of effort to maintain, making it all too easy to fat-finger something. Accidental exposure is a massive risk that could all too easily result in security vulnerabilities, data corruption or loss of trade secrets. Your features may not be as isolated from each other as you thought they were, and you may end up deploying bugs to your production environment. I’m not speaking theoretically here either: this has actually happened to me. Unless your feature toggles are well bedded in, and you have some pretty mature processes and infrastructure in place to manage them, you’re asking for trouble. Certainly, my own experience of them has been uniformly negative.

But wait! He actually goes on to acknowledge these concerns, albeit in drastically watered down and sugar-coated terms compared to the sheer FUD that gets spouted against feature branches:

The traditional concern with that approach is that you’ve got a half finished feature in your codebase, so OK it’s not visible in the UI, but you made some changes, surely you may have broken something? And the right response to that is to have confidence in your automated testing that you should also have, to guarantee that even with a half-finished feature that’s hidden with a feature toggle, you haven’t broken anything else.

Right. So, after having given us no guidance whatsoever on how we can use feature branches responsibly and effectively, they then go on to give us guidance on how to responsibly use what is, when it boils right down to it, a high-risk, high-maintenance hack. If that isn’t a lopsided view of the subject, then what is?

In fact, they go on to talk about one case where the client had so many feature toggles that they needed to have their own custom compilation of the Linux kernel just to handle them all. But oddly enough, they don’t translate that into “feature toggles considered harmful” in the way they do for feature branches.

For what it’s worth, this highlights a problem with the very idea of Continuous Integration itself.

Continuous Integration is largely a workaround anyway.

It seems that all the FUD about feature branches boils down to one thing: we should restrict ourselves to trunk-based development because Continuous Integration is the One True Way to do configuration management. But let’s just take a step back and ask ourselves why we do Continuous Integration in the first place? Largely because we were restricted to trunk-only development. If you check in code that breaks the build, then go home, and then someone else checks it out, they can’t get anything done till you return and fix it. You constantly need to have a ready-to-deploy version of your code in case of security vulnerabilities. While this isn’t the whole picture, and there are other arguments in favour of Continuous Integration, it is at least partly a hack to work around the restrictions of trunk-based development. The vagueness of Martin Fowler’s “cool rule of thumb”—that you should check in “at least once a day” is testimony to this. Cargo culting your way through advice like that will lead to you checking in incomplete, buggy or even downright broken code, and the need for high-maintenance hacks such as feature toggles to compensate.

A hack to work around the limitations of another workaround for the limitations of our tooling. All this being lauded as a best practice. What on earth is the world coming to?

But these restrictions don’t apply to DAG-based tooling, where you can easily base your work off the last known good revision. Yes, you can end up with conflicts when you branch and merge, and yes, you can easily shoot yourselves in the foot if you’re not careful, and yes, you can end up with Big Scary Merges, but with good communication and planning among developers, many teams can—and do—easily and confidently handle surprisingly long-running branches with few if any problems. And yes, you should be keeping your work items as small as possible so that you can integrate as frequently as possible. Yes, feature branches get misused. But the correct response to misuse is not disuse, but proper use.

The most dangerous factor in this matter, however, is Martin Fowler’s reputation. Many agilists view him as infallible when speaking ex cathedra on matters of architecture and methodology, and for him to promote points of view such as these is to hand ammunition to the anti-DVCS lobby on a plate. Were this to lead to a wholescale rejection of feature branching and distributed source control, it would set the entire agile movement back several years, and that would be very sad.

What is really needed is a holistic, unbiased review of both feature branches and Continuous Integration, to see how we can use feature branches responsibly, and at the same time how Continuous Integration and Continuous Delivery can best be adapted to take them into account. But simply insisting on hanging onto the established ways of doing things with complete disregard for new and better technology is cargo cult programming, and it certainly isn’t agile.