james mckay dot net
because there are few things that are less logical than business logic

Why does Martin Fowler not understand feature branches?

Update (September 2018): This post is now seven years old and no longer represents my current thinking on the feature branches versus feature toggles debate. For an updated and more nuanced view, see my January 2017 update: Martin Fowler and feature branches revisited.

Update (July 2011): this subject seems to have generated quite a lively discussion in the blogosphere and on Twitter. If you want to see what people are saying elsewhere and follow the discussion further, here are some links:

Make sure you read the comments on these posts to get both sides of the story.


In the software development world in general, and in the agile community in particular, Martin Fowler is pretty much considered the ultimate authority on just about everything. He’s not like Joel Spolsky, who preaches a feel-good message laced with linkbait such as you should invent your own programming language but not use Ruby, or that the SOLID principles are too bureaucratic to be useful. He speaks in measured tones based on years of experience working with large enterprise clients. He (mostly) shuns controversy and focuses on value. He adopts a Neutral Point Of View. He is not just an agile coach, or an agile luminary, he is one of the authors of the Agile Manifesto itself. He has written several ground-breaking books on software engineering, including Refactoring and Patterns of Enterprise Application Architecture. Someone of that stature makes the best of us look like schoolboys in short trousers by comparison.

So when I came across this video of him and his colleague Mike Mason speaking on the perils of feature branches, I was absolutely horrified. Horrified, because much of what they have to say is totally misleading and completely misrepresents what branch-per-feature is all about. It sounds like the kind of anti-branch FUD that I only ever expected to hear from Highly Paid TFS Consultants and vendors of Subversion-based software, who have the most to lose from the next generation of source control tools that make branching and merging safe, robust and easy.

Note first of all what they’re saying.

  • They are saying that Feature Branching is incompatible with Continuous Integration.
  • They are not saying that you can shoot yourself in the foot if you do Feature Branching wrong, but you can be much more productive if you do Feature Branching right. They are saying that Feature Branching is bad. Period.
  • They are not saying that some tools aren’t cut out for Feature Branching but others are, and if you’re stuck with one of the ones that aren’t, here’s how to make the most of it. They’re saying that Feature Branching is bad. Period.

Now maybe they may argue that they don’t actually mean that, or that I’ve misunderstood them, but the problem is that that is what the anti-DVCS scaremongers will hear, and as sure as eggs are eggs, they will milk it for all it’s worth. Certainly, the whole tone of the video towards feature branches is strongly negative. And in some places, factually inaccurate. Besides which, the alternative that they propose is no better.

Feature branches are not about sporadic integration.

Fowler contrasts feature branching with Continuous Integration as if the two are diametrically opposed to each other. He says this:

The cool rule of thumb that I use for Continuous Integration is that everybody’s integrating back to the mainline at least once a day, while with feature branching, people can go for many days, possibly many weeks, until they complete the feature, and then they only integrate once the feature is done. And it’s that frequency of integration which is the key difference.

No, no, no, no, no, no, no! That is not what feature branches are about!

Feature branches are not about increasing the amount of time between integration, they are about decreasing the time between check-ins. You still break down your features into the smallest tasks you can deliver. You still integrate as frequently as you can, yes, at least once a day where possible. The difference with feature branches is that you start checking in code several times an hour. Why? First, this gives you checkpoints that you can roll back to if you discover that you’ve made a mistake, and second, it makes the thinking behind what you’re doing easier to describe and decipher.

The reason why people think feature branches are for long-running epics is because that is the way that old school centralised tools have taught them to think. They’ve made branching and merging hard through poor tooling, algorithm support and architecture, so people do everything in trunk except for epics, then they run into problems and the whole pattern becomes self-reinforcing. Whereas the correct way to do it is to make branching and merging the default option, so people start off with small, frequent merges, and naturally expand into epics as they develop the maturity to handle them.

Feature branches versus Continuous Integration is a false dichotomy. You don’t do either/or, you do both/and.

Semantic conflicts are not specific to branching and merging.

Fowler then goes on to bring up the issue of semantic conflicts.

And the issue here is, yes they can make the problems of textual merging go away, and certainly the textual merging algorithms have improved greatly over the last ten or twenty years, but there’s still semantic issues that go deeper than the pure text, and they, those do not go away, and as a result you still have big, painful merges and the pain increases exponentially the longer you leave between integrations. So by integrating more frequently, even though you’re doing a lot more integrating, the integrations are sufficiently small that they’re not actually causing a lot of problems and as a result you can do them a lot more smoothly as you go along.

I’m sorry, but that is a straw man. Yes, semantic conflicts are a problem, but they are not specific to branching and merging. Any kind of integration will have semantic conflicts. Trunk-based development will have semantic conflicts. A lock/edit/unlock model such as that adopted by Visual SourceSafe will have semantic conflicts. Everyone editing the same files directly on the same server and not using source control at all will have semantic conflicts. Semantic conflicts are a fact of life no matter what integration model you use. Get used to it.

Besides, the integrations where you need to worry most about semantic conflicts are not the biggest, scariest ones. By the time you get to that level, you are looking at textual conflicts all over the place anyway, so you’ll be on your guard and double checking everything. The semantic conflicts you need to worry about are the ones where there are no textual conflicts at all, where you have dropped your guard. In other words, the size of merges that are typical of Continuous Integration. In any case, we have tools to deal with them. They are called compilers, static analysis, and unit tests. As for any that slip through that net, all you can do is treat them like any other bugs. The question here is, how many of these semantic conflicts translate into new bugs and regressions in your code compared to bugs introduced through any other means? When you are used to feature branching, that is exactly how you view semantic conflicts: as no greater or less a deal than any other form of bug.

For what it’s worth, semantic conflicts have been the subject of some academic research, for example, in this paper, which is well worth a read for anyone who is concerned about them, their prevalence, the risks that they may or may not pose, and strategies to mitigate their effects. But it would be naive and wrong to be paranoid about feature branches because of them, because, as I said, they can crop up just as easily in trunk-based development.

Feature toggles can cause problems too. Bigger, scarier problems.

Feature toggles are a way of avoiding branching and merging, which Joel Spolsky describes as “a workaround for the fact that your version control tool is not doing what it’s meant to do.” Branch by Abstraction is much the same thing, except that it uses IOC containers instead of if statements. Mike Mason explains it as follows:

So, feature toggles are a mechanism whereby you implement a new feature, but simply have it switched off, usually in the software configuration, and in most systems that can be as simple as not including a feature in the UI, so it’s actually being developed under the covers, it’s present in the source code, but you don’t enable it in the UI so nobody actually sees it.

You see what the problem is here? And man, it is a massive problem—visible or not, you are still deploying code into production that you know for a fact to be buggy, untested, incomplete, and quite possibly incompatible with your live data. Your if statements and configuration settings are themselves code which is subject to bugs—and furthermore can only be tested properly in production. They are also a lot of effort to maintain, making it all too easy to fat-finger something. Accidental exposure is a massive risk that could all too easily result in security vulnerabilities, data corruption or loss of trade secrets. Your features may not be as isolated from each other as you thought they were, and you may end up deploying bugs to your production environment. I’m not speaking theoretically here either: this has actually happened to me. Unless your feature toggles are well bedded in, and you have some pretty mature processes and infrastructure in place to manage them, you’re asking for trouble. Certainly, my own experience of them has been uniformly negative.

But wait! He actually goes on to acknowledge these concerns, albeit in drastically watered down and sugar-coated terms compared to the sheer FUD that gets spouted against feature branches:

The traditional concern with that approach is that you’ve got a half finished feature in your codebase, so OK it’s not visible in the UI, but you made some changes, surely you may have broken something? And the right response to that is to have confidence in your automated testing that you should also have, to guarantee that even with a half-finished feature that’s hidden with a feature toggle, you haven’t broken anything else.

Right. So, after having given us no guidance whatsoever on how we can use feature branches responsibly and effectively, they then go on to give us guidance on how to responsibly use what is, when it boils right down to it, a high-risk, high-maintenance hack. If that isn’t a lopsided view of the subject, then what is?

In fact, they go on to talk about one case where the client had so many feature toggles that they needed to have their own custom compilation of the Linux kernel just to handle them all. But oddly enough, they don’t translate that into “feature toggles considered harmful” in the way they do for feature branches.

For what it’s worth, this highlights a problem with the very idea of Continuous Integration itself.

Continuous Integration is largely a workaround anyway.

It seems that all the FUD about feature branches boils down to one thing: we should restrict ourselves to trunk-based development because Continuous Integration is the One True Way to do configuration management. But let’s just take a step back and ask ourselves why we do Continuous Integration in the first place? Largely because we were restricted to trunk-only development. If you check in code that breaks the build, then go home, and then someone else checks it out, they can’t get anything done till you return and fix it. You constantly need to have a ready-to-deploy version of your code in case of security vulnerabilities. While this isn’t the whole picture, and there are other arguments in favour of Continuous Integration, it is at least partly a hack to work around the restrictions of trunk-based development. The vagueness of Martin Fowler’s “cool rule of thumb”—that you should check in “at least once a day” is testimony to this. Cargo culting your way through advice like that will lead to you checking in incomplete, buggy or even downright broken code, and the need for high-maintenance hacks such as feature toggles to compensate.

A hack to work around the limitations of another workaround for the limitations of our tooling. All this being lauded as a best practice. What on earth is the world coming to?

But these restrictions don’t apply to DAG-based tooling, where you can easily base your work off the last known good revision. Yes, you can end up with conflicts when you branch and merge, and yes, you can easily shoot yourselves in the foot if you’re not careful, and yes, you can end up with Big Scary Merges, but with good communication and planning among developers, many teams can—and do—easily and confidently handle surprisingly long-running branches with few if any problems. And yes, you should be keeping your work items as small as possible so that you can integrate as frequently as possible. Yes, feature branches get misused. But the correct response to misuse is not disuse, but proper use.

The most dangerous factor in this matter, however, is Martin Fowler’s reputation. Many agilists view him as infallible when speaking ex cathedra on matters of architecture and methodology, and for him to promote points of view such as these is to hand ammunition to the anti-DVCS lobby on a plate. Were this to lead to a wholescale rejection of feature branching and distributed source control, it would set the entire agile movement back several years, and that would be very sad.

What is really needed is a holistic, unbiased review of both feature branches and Continuous Integration, to see how we can use feature branches responsibly, and at the same time how Continuous Integration and Continuous Delivery can best be adapted to take them into account. But simply insisting on hanging onto the established ways of doing things with complete disregard for new and better technology is cargo cult programming, and it certainly isn’t agile.

3 trackbacks:

15 comments:

  • # Reply from Adam D. at 17:52 on 7 Jul 2011

    Very well said.

  • # Reply from Eric Duncan at 23:19 on 7 Jul 2011

    Absolutely. Daily merges from the always-in-a-releasable-state $trunk into your feature branch keeps you always in check, for your reintegration back into the trunk line after QA.

    Very very well said.

  • # Reply from Martinos at 00:40 on 8 Jul 2011

    Great article, I can’t agree more. For me, feature toggles are branches within source code. It can be worse that feature branches since we have no tool to merge them back (clean them out).

    I think an experienced programmer will find the right balance between Continuous Integration and Feature Branching. If it’s the Feature Branch owner responsibility to merge it back to the mainline he will tend to merge it as often as he can.

    With DVC you can at least make continuous integration from the master to your feature branch, this way it will facilitate the merge to the mainline.

  • # Reply from Mark Thomas at 11:11 on 8 Jul 2011

    You seem to be preaching to the DVCS choir. For those of us who are Fowler followers but not DVCS aficionados, I’m not sure you’ve provided a compelling counterargument. My mind is open. Convince me.

  • # Reply from Jean Helou at 15:41 on 8 Jul 2011

    @Mark Thomas, consider martinos’ insightful comment:
    “feature toggle are branches within the code” you can’t disagree with that can you.
    Now the feature toggle pattern also states that you should keep your toggles to a minimum, cleaning up and removing the old “branches” when they are
    no longer used.
    Now for new features it is pretty easy but when altering an existing feature where the effects are interleaved throughout a dozen or more of classes it becomes fairly painful to go through all the classes again to remove all the “ifs”.
    Tools can help like spring3.1 profiles I guess but again you have to manually drop dead code after the fact, with the risk of removing too much or leaving the deadcode in place.

    I had this exact problem recently, I considered using feature toggle and In the end decided against it because of the maintenance risk.
    Feature toogle also has other side effects like artificially increasing cyclomatic complexity

  • # Reply from Martinos at 16:22 on 8 Jul 2011

    Mark I agree with you that all this discussion depends of you tooling. If you are stucked with a centralized source control (a lot of companies are) feature branching can be a pain.

    There is another thing that I would like to mention is that in other to achieve continuous integration you MUST have tests. Which is not always the case. Don’t get me wrong I am a big fan of automated testing. If you don’t have test (let say for legacy reason), feature branching is almost necessary for big features.

    For me there is a rule of thumb in the case that you have good test coverage:
    If you have a centralized source control: avoid feature branch at all cost.
    If you have a DVCS, use feature branch but merge has often has possible to the mainline.

  • # Reply from Perryn Fowler at 02:03 on 11 Jul 2011

    Hi James, I have a different view to you

    1) To me, Feature branching IS diametrically opposed to CI, because CI is about integration *continuously* while feature branching is about only integrating when your feature is finished.

    You make the valid point that if a ‘feature’ is small enough then there is little practical difference. However, one of the major drivers of both Feature Branching and Feature Toggles is the ability to release without having half finished features showing. This means that the size of your feature ( and hence the length of the life of your branch) is tied to the size of a feature that makes sense to a user, not what you can do in a day.

    Hence, in my experience at least, feature branches have always resulted in integration happening much less frequently than once a day. Personally I prefer to have integration happening much *more* often than once a day.

    2) True semantic conflicts are not specific to branching and merging, however branching and merging does delay their discovery, leading to slower feedback.

    without branching, if I try to do something that results in a semantic conflict with something someone else is doing then I find out almost immediately. With branching I only find out when we both merge – by that stage its often harder to remember the exact changes made, and we have both built more code on our assumptions. This makes the mess harder to sort out.

    3) True, feature toggles done badly can be bad, just as feature branches done badly can be bad. To me, the difference is that I know of no way to avoid the things I talked about in 1 & 2 with feature branching. Perhaps you do?

    4) To me, CI is not a hack to work around the problems with trunk based development, it is
    a tool to *enable* trunk based development. The reasons I *want* to do trunk based development are largely around point 2.

    5) Lastly, I must take issue with your attitude that Martin should not publicly state his considered advice because people respect and listen to him. With that approach, you and I would have to keep our opinions to ourselves if we ever become as respected as Martin. Does that make any sense?

  • # Reply from James at 09:36 on 11 Jul 2011

    No, it’s perfectly OK for him to express his opinions. It’s just that he’s come down firmly on one side of a pretty controversial topic, which is unusual for him. I just wanted to make the point that there is another side to the argument.

    As to your other points: they say the proof of the pudding is in the eating, and there are loads of DVCS teams out there who are successfully managing feature branches of all shapes and sizes without finding conflicts, textual or semantic, to be a serious problem. Having said that, some people are less successful. I’ve a follow-up post in preparation that examines why in more detail.

  • # Reply from pablo at 21:48 on 12 Jul 2011

    Hi there!

    I absolutely agree with you! There are a huge number of common misconceptions and I think you really highlight most of them, starting with the “semantic conflicts”. People tend to think that no branching means no merging, and that’s totally untrue.

    The good news is that time is on “our” side, I mean, on the side of “feature branches”, just take a look at Git, Mercurial or what we’re doing with Plastic SCM.

    Great article!

    pablo

  • # Reply from pablo at 21:51 on 12 Jul 2011

    @Martinos:

    “If you have a centralized source control” -> well, I can’t agree here. The point is not being DVCS or not, the point is having proper merge tracking or not. Good-ol ClearCase was able to do proper branching eons ago… 😉

    Our first release of Plastic SCM, 5 years ago, wasn’t distributed but was able to do feature branches pretty well… 🙂

    But yes, when we say “dvcs” we really say “modern SCMs”, and then that’s true: proper modern SCM tooling can handle feature branches, while the old “centralized ones” (especially SVN) can’t.

  • # Reply from Martinos at 22:26 on 13 Jul 2011

    @pablo: I have never worked with Clear Case, so I can argue with you. In fact instead of talking about DVCS I should talk about source controls that allow private local branches.

    Maybe I have a misconception of feature branch. In fact, when I develop I often create a branch that reflect an atomic change, which is functional as a whole. It might not be a feature, it might be a bug fix or a refactoring. This kind of branch might contain only one commit. I call it a feature branch, but it might not be the proper term.

    Maybe when Martin Fowler talks about a Feature branch he talks about a branch that contains a full blown feature, which would be logical.

    But how are called those branch that we create for those small changes that are not features ?

  • # Reply from James Brechtel at 05:13 on 16 Jul 2011

    Hi James,

    First of all there seems to be an issue of definition here. You start out by saying “You still integrate as frequently as you can, yes, at least once a day where possible.” and from there you seem to respond to the rest of what was said in the video as if this is what Martin+Mike were talking about.* Instead Martin+Mike were talking about a scenario where you have long lived branches (~2-3 days normally in my experience but sometimes a week or more) that are NOT merged back to trunk until their ‘feature’ is complete. These are two *completely* different scenarios with wildly different impacts to practices such as continuous integration and refactoring. Perhaps the problem is simply there isn’t a commonly agreed upon meaning of what ‘feature branching’ is?

    Also, regarding your definition of feature branching, why is this a ‘feature branch’ at all if you’re merging to trunk at least once a day? Couldn’t you just make local commits until you’re ready to push to trunk? I know developers who do this but have never heard it characterized as ‘feature branching’ but it sounds like the workflow you’re describing.

    (Full disclosure, I work for ThoughtWorks)

    *the feature toggles bit aside (I’m not a fan of them as a common practice)

  • # Reply from Christophe at 10:51 on 4 Sep 2011

    As James Brechtel is saying, this seems to be an issue with the definition of what feature branching is. When I saw the video (before reading your comments), I understood Martin was talking about companies that still develop features away from the mainline, sometimes to avoid compatibility issues and end up staying far longer than expected away from the mainline and having to do hell integration of months: this is still common through some companies. This is an old pattern that still exists unfortunately.
    In a nutshell, it does not seem you are talking about the same thing.

  • # Reply from Jeena at 08:31 on 15 Sep 2011

    Correct, absolutely sharable.
    I completely agree with your post.

Comments are closed.