Why does Martin Fowler not understand feature branches?

Update (September 2018): This post is now seven years old and no longer represents my current thinking on the feature branches versus feature toggles debate. For an updated and more nuanced view, see my January 2017 update: Martin Fowler and feature branches revisited.

Update (July 2011): this subject seems to have generated quite a lively discussion in the blogosphere and on Twitter. If you want to see what people are saying elsewhere and follow the discussion further, here are some links:

• ThoughtWorks consultant Sarah Taraporewalla has reported her experience with feature branches and feature toggles.
• Adam Dymitruk has a post giving advice on how to make feature branches work for you.
• Jez Humble, author of Continuous Delivery, weighs in with his own point of view.

Make sure you read the comments on these posts to get both sides of the story.

---

In the software development world in general, and in the agile community in particular, Martin Fowler is pretty much considered the ultimate authority on just about everything. He’s not like Joel Spolsky, who preaches a feel-good message laced with linkbait such as you should invent your own programming language but not use Ruby, or that the SOLID principles are too bureaucratic to be useful. He speaks in measured tones based on years of experience working with large enterprise clients. He (mostly) shuns controversy and focuses on value. He adopts a Neutral Point Of View. He is not just an agile coach, or an agile luminary, he is one of the authors of the Agile Manifesto itself. He has written several ground-breaking books on software engineering, including Refactoring and Patterns of Enterprise Application Architecture. Someone of that stature makes the best of us look like schoolboys in short trousers by comparison.

So when I came across this video of him and his colleague Mike Mason speaking on the perils of feature branches, I was absolutely horrified. Horrified, because much of what they have to say is totally misleading and completely misrepresents what branch-per-feature is all about. It sounds like the kind of anti-branch FUD that I only ever expected to hear from Highly Paid TFS Consultants and vendors of Subversion-based software, who have the most to lose from the next generation of source control tools that make branching and merging safe, robust and easy.

Note first of all what they’re saying.

• They are saying that Feature Branching is incompatible with Continuous Integration.
• They are not saying that you can shoot yourself in the foot if you do Feature Branching wrong, but you can be much more productive if you do Feature Branching right. They are saying that Feature Branching is bad. Period.
• They are not saying that some tools aren’t cut out for Feature Branching but others are, and if you’re stuck with one of the ones that aren’t, here’s how to make the most of it. They’re saying that Feature Branching is bad. Period.

Now maybe they may argue that they don’t actually mean that, or that I’ve misunderstood them, but the problem is that that is what the anti-DVCS scaremongers will hear, and as sure as eggs are eggs, they will milk it for all it’s worth. Certainly, the whole tone of the video towards feature branches is strongly negative. And in some places, factually inaccurate. Besides which, the alternative that they propose is no better.

Feature branches are not about sporadic integration.

Fowler contrasts feature branching with Continuous Integration as if the two are diametrically opposed to each other. He says this:

The cool rule of thumb that I use for Continuous Integration is that everybody’s integrating back to the mainline at least once a day, while with feature branching, people can go for many days, possibly many weeks, until they complete the feature, and then they only integrate once the feature is done. And it’s that frequency of integration which is the key difference.

No, no, no, no, no, no, no! That is not what feature branches are about!

Feature branches are not about increasing the amount of time between integration, they are about decreasing the time between check-ins. You still break down your features into the smallest tasks you can deliver. You still integrate as frequently as you can, yes, at least once a day where possible. The difference with feature branches is that you start checking in code several times an hour. Why? First, this gives you checkpoints that you can roll back to if you discover that you’ve made a mistake, and second, it makes the thinking behind what you’re doing easier to describe and decipher.

The reason why people think feature branches are for long-running epics is because that is the way that old school centralised tools have taught them to think. They’ve made branching and merging hard through poor tooling, algorithm support and architecture, so people do everything in trunk except for epics, then they run into problems and the whole pattern becomes self-reinforcing. Whereas the correct way to do it is to make branching and merging the default option, so people start off with small, frequent merges, and naturally expand into epics as they develop the maturity to handle them.

Feature branches versus Continuous Integration is a false dichotomy. You don’t do either/or, you do both/and.

Semantic conflicts are not specific to branching and merging.

Fowler then goes on to bring up the issue of semantic conflicts.

And the issue here is, yes they can make the problems of textual merging go away, and certainly the textual merging algorithms have improved greatly over the last ten or twenty years, but there’s still semantic issues that go deeper than the pure text, and they, those do not go away, and as a result you still have big, painful merges and the pain increases exponentially the longer you leave between integrations. So by integrating more frequently, even though you’re doing a lot more integrating, the integrations are sufficiently small that they’re not actually causing a lot of problems and as a result you can do them a lot more smoothly as you go along.

I’m sorry, but that is a straw man. Yes, semantic conflicts are a problem, but they are not specific to branching and merging. Any kind of integration will have semantic conflicts. Trunk-based development will have semantic conflicts. A lock/edit/unlock model such as that adopted by Visual SourceSafe will have semantic conflicts. Everyone editing the same files directly on the same server and not using source control at all will have semantic conflicts. Semantic conflicts are a fact of life no matter what integration model you use. Get used to it.

Besides, the integrations where you need to worry most about semantic conflicts are not the biggest, scariest ones. By the time you get to that level, you are looking at textual conflicts all over the place anyway, so you’ll be on your guard and double checking everything. The semantic conflicts you need to worry about are the ones where there are no textual conflicts at all, where you have dropped your guard. In other words, the size of merges that are typical of Continuous Integration. In any case, we have tools to deal with them. They are called compilers, static analysis, and unit tests. As for any that slip through that net, all you can do is treat them like any other bugs. The question here is, how many of these semantic conflicts translate into new bugs and regressions in your code compared to bugs introduced through any other means? When you are used to feature branching, that is exactly how you view semantic conflicts: as no greater or less a deal than any other form of bug.

For what it’s worth, semantic conflicts have been the subject of some academic research, for example, in this paper, which is well worth a read for anyone who is concerned about them, their prevalence, the risks that they may or may not pose, and strategies to mitigate their effects. But it would be naive and wrong to be paranoid about feature branches because of them, because, as I said, they can crop up just as easily in trunk-based development.

Feature toggles can cause problems too. Bigger, scarier problems.

Feature toggles are a way of avoiding branching and merging, which Joel Spolsky describes as “a workaround for the fact that your version control tool is not doing what it’s meant to do.” Branch by Abstraction is much the same thing, except that it uses IOC containers instead of if statements. Mike Mason explains it as follows:

So, feature toggles are a mechanism whereby you implement a new feature, but simply have it switched off, usually in the software configuration, and in most systems that can be as simple as not including a feature in the UI, so it’s actually being developed under the covers, it’s present in the source code, but you don’t enable it in the UI so nobody actually sees it.

You see what the problem is here? And man, it is a massive problem—visible or not, you are still deploying code into production that you know for a fact to be buggy, untested, incomplete, and quite possibly incompatible with your live data. Your if statements and configuration settings are themselves code which is subject to bugs—and furthermore can only be tested properly in production. They are also a lot of effort to maintain, making it all too easy to fat-finger something. Accidental exposure is a massive risk that could all too easily result in security vulnerabilities, data corruption or loss of trade secrets. Your features may not be as isolated from each other as you thought they were, and you may end up deploying bugs to your production environment. I’m not speaking theoretically here either: this has actually happened to me. Unless your feature toggles are well bedded in, and you have some pretty mature processes and infrastructure in place to manage them, you’re asking for trouble. Certainly, my own experience of them has been uniformly negative.

But wait! He actually goes on to acknowledge these concerns, albeit in drastically watered down and sugar-coated terms compared to the sheer FUD that gets spouted against feature branches:

The traditional concern with that approach is that you’ve got a half finished feature in your codebase, so OK it’s not visible in the UI, but you made some changes, surely you may have broken something? And the right response to that is to have confidence in your automated testing that you should also have, to guarantee that even with a half-finished feature that’s hidden with a feature toggle, you haven’t broken anything else.

Right. So, after having given us no guidance whatsoever on how we can use feature branches responsibly and effectively, they then go on to give us guidance on how to responsibly use what is, when it boils right down to it, a high-risk, high-maintenance hack. If that isn’t a lopsided view of the subject, then what is?

In fact, they go on to talk about one case where the client had so many feature toggles that they needed to have their own custom compilation of the Linux kernel just to handle them all. But oddly enough, they don’t translate that into “feature toggles considered harmful” in the way they do for feature branches.

For what it’s worth, this highlights a problem with the very idea of Continuous Integration itself.

Continuous Integration is largely a workaround anyway.

It seems that all the FUD about feature branches boils down to one thing: we should restrict ourselves to trunk-based development because Continuous Integration is the One True Way to do configuration management. But let’s just take a step back and ask ourselves why we do Continuous Integration in the first place? Largely because we were restricted to trunk-only development. If you check in code that breaks the build, then go home, and then someone else checks it out, they can’t get anything done till you return and fix it. You constantly need to have a ready-to-deploy version of your code in case of security vulnerabilities. While this isn’t the whole picture, and there are other arguments in favour of Continuous Integration, it is at least partly a hack to work around the restrictions of trunk-based development. The vagueness of Martin Fowler’s “cool rule of thumb”—that you should check in “at least once a day” is testimony to this. Cargo culting your way through advice like that will lead to you checking in incomplete, buggy or even downright broken code, and the need for high-maintenance hacks such as feature toggles to compensate.

A hack to work around the limitations of another workaround for the limitations of our tooling. All this being lauded as a best practice. What on earth is the world coming to?

But these restrictions don’t apply to DAG-based tooling, where you can easily base your work off the last known good revision. Yes, you can end up with conflicts when you branch and merge, and yes, you can easily shoot yourselves in the foot if you’re not careful, and yes, you can end up with Big Scary Merges, but with good communication and planning among developers, many teams can—and do—easily and confidently handle surprisingly long-running branches with few if any problems. And yes, you should be keeping your work items as small as possible so that you can integrate as frequently as possible. Yes, feature branches get misused. But the correct response to misuse is not disuse, but proper use.

The most dangerous factor in this matter, however, is Martin Fowler’s reputation. Many agilists view him as infallible when speaking ex cathedra on matters of architecture and methodology, and for him to promote points of view such as these is to hand ammunition to the anti-DVCS lobby on a plate. Were this to lead to a wholescale rejection of feature branching and distributed source control, it would set the entire agile movement back several years, and that would be very sad.

What is really needed is a holistic, unbiased review of both feature branches and Continuous Integration, to see how we can use feature branches responsibly, and at the same time how Continuous Integration and Continuous Delivery can best be adapted to take them into account. But simply insisting on hanging onto the established ways of doing things with complete disregard for new and better technology is cargo cult programming, and it certainly isn’t agile.