As a result of design decisions made early on, and a firm commitment to backwards compatibility, the project – at least until a 2.0 release – is locked in to its limitations. Merging, and more specifically merge tracking, has been a huge pain in the backside for large numbers of users.

The 1.5 release was mostly a lateral move from the svnmerge.py project, and subsequent releases did little more than patch a few of the more glaring holes in the original implementation, although conflict handling has seen dramatic improvements.

So how do we play into Subversion’s strengths without playing into its weaknesses? My answer to that is 4-fold:

1) Merge strategy

It may seem backwards to talk about this first rather than branching, but merging is way more complicated than branching, and so the peculiarities and limitations of merging in SVN may very well dictate you branch strategy, so I feel it’s important to understand the parameters of your options in this area, make the appropriate choices, and work backwards from there.

Merge tracking in SVN, although much improved over past releases, is still very much a work in progress. Your desired approach relies heavily on SVN’s ability to keep track of cherry-picked merges. To keep from hitting SVN’s current limitations, you’ll want to keep your merges very straight-forward, and be VERY careful to understand exactly what you’re doing when deviating from the model I’m outlining. SVN is very flexible and will let you get into all kinds of situations you’re not going to want to be in without adult supervision. The smart thing to do is to constantly ask yourself “I could do it this way, but should I?”

For example: you can merge files individually, but SVN is built around atomic revisions that generally contain changes to multiple files. SVN’s merge tracking feature attempts to keep track of partial merges of revisions, but is imperfect in this regard, so to avoid losing track of what has and hasn’t been merged, you should:

a) educate developers to not commit changes related to multiple issues at once
b) only ever merge entire revisions. I can’t stress this enough: never merge partial revisions.
c) if issues are or do happen to become legitimately intertwined, track them accordingly and not allow them to be merged/built/released independently.

2) Branch strategy

Now that we’ve established the parameters for how to keep merges manageable, the rest is fairly straight-forward in that it’s mostly dictated by this merge strategy.

You create a branch for a limited set of changes (project, set of bugs/enhancements, whatever the case may be). This in and of itself is smart. The concern is obviously that “what if I don’t want to merge/release/build everything on the branch, but rather a subset of these changes?”. The answer to that is – as is often the case – “it depends.”

You can either:

a) create a temporary integration branch (based on trunk), merge the changes you do want to it from the original branch, resolve any integration issues, and merge changes back to your trunk the same way you would otherwise. Scrap the integration branch (archive or remove) when it’s served its purpose. Branches are version-controlled in SVN, so you can always go back and restore one if need be.
b) selectively merge revisions from your branch back to trunk and resolve any integration issues on trunk.
c) merge trunk changes to your branch, resolve integration issues on the branch, and reintegrate your branch with trunk.

Following this model, you are never more than 1 step removed from trunk (i.e. you’re either on a trunk or on a branch, and a branch is always created from trunk), and changes never get merged more than twice (i.e. either from trunk to branch, from branch to trunk, or from branch to integration branch to trunk).

Depending on the scenario (changes in question, whether the merge is partial or complete) you *should* be going in one of these three directions. The important thing to remember, is that whenever you merge trunk changes into the branch, you should ALWAYS reintegrate that branch in its entirety. Of course, once you understand the rule, you can break the rule, but I wouldn’t go down that road unless I had an expert at SVN/SCM to manage the heavy lifting it will almost certainly involve, either immediately or eventually.

3) Hire an expert

Most companies still look at build/SCM work as something to volunteer a junior developer for. Few decisions will get you into trouble faster, or limit the effectiveness of the position more, than going this route. Build/configuration/release management is a skill set and I would even argue an art form all on its own. I cannot stress enough the value the right resource in that chair will bring to the table. The time and money saved, not having to deal with the immediate and ripple effects of issues that shouldn’t or don’t need to be, can save a project, or even a business.

4) Tools

Possibly the most important tool in all of this is what you use to keep track of all the information involved in doing the above and not miss something along the way. You’ll want to be using a tool that can

a) keep track of your issues (should be obvious, right?)
b) keep track of the relationships between your issues and use these relationships to automate notifications to developers when upstream issues change, possibly affecting downstream dependent issues (this eliminates the bulk of tools out there)
c) relate SVN revisions to issues
d) provide top-to-bottom traceability across all of the above

First and foremost, an organization’s choice to make build management a priority is a strategic one. So the role has to be filled by someone who can think and operate at that level, as well as get their hands dirty and do the work on a tactical level. It’s a careful balance of big picture vs. “stuff needs to get done today”. No matter how good you are, you’re not going to be able to do it all at once, all of the time, so planning and managing your time is critical, especially if you’re juggling a lot of different projects at once.

These are some of the key lessons I learned along the way.

Not too much, not too fast

Give yourself time to get your bearings and make sure you really understand the environment you’re in. Depending on experience and the complexity of the situation, this can take days or even weeks.

Once you feel ready to dive in, limit yourself to one project, or account, or application, and find something you can do in no more than a day, something that isn’t going to start a domino effect you won’t be able to control. The key concept here is bite-size. Whenever you notice something that just doesn’t look or feel right, but it’s too big or tangled up to tackle right then & there, take notes. When you run out of things to fix, go back to your list and find something new to sink your teeth into. Make a habit of this. It’s a great way to stay fresh and productive once you have everything working just right and things just run along smoothly. The worst thing you can ever do is nothing. Slow is smooth, smooth is fast.

Change is hard

Humans are creatures of habit. Most people aren’t wired for rapidly changing, dynamic environments. And most people that do will generally prefer a solid foundation of routine for balance. The decision to start investing in build management can only mean that there’s going to be some changes, and they can fundamentally affect the way people do their jobs, and how and to what extent they’re going to be held personally accountable for the quality of their work. I find that the trick is to just start changing small, seemingly innocuous things, and pretty soon teams are used to the new routine of continuous improvement. The secret to making this work is to not screw up. If you are occasionally human and stumble, eat your humble pie, fix what you broke and make sure you don’t make the same mistake twice. You’ll be surprised at how much you can get away with if you follow these simple steps.

If you can script it, you shouldn’t be doing it

Nothing destroys the reliability of any process faster than having it be manual. People make mistakes, to expect them not to is fundamentally unfair. If your process is going to be predictable, repeatable, reliable, and fast, it simply has to be automated. It’s also the only way you’re going to be able to implement continuous integration, continuous deployment, continuous anything.

If you can’t script it, you’re not doing it right

See above. I can’t stress enough how important this is.

If it breaks once, something automated should catch it if it happens again

There’s an old Dutch saying that roughly translates to: “even a donkey generally doesn’t knock into the same rock twice.” There are few things that will ruin my day faster than running into the same problem twice. Proponents of test-driven development will be familiar with the concept: if something breaks, write a test that catches it, fix the problem, and now your test passes. If the problem ever regresses, the test should fail again, and you catch it before it gets any further.

In production environments, operational issues should be caught and reported on by monitoring tools such as Nagios. Again, if it happens once, find out how it could and should have been caught sooner, and put the necessary controls in place.

It probably wasn’t your fault, but it is your problem to fix

Being the bottleneck means everything the team does crosses your desk at some point. Generally the only other person on the project who touches everything is the project manager, but he or she is most likely not nearly as technical as you, and won’t be able to zero in on the root cause nearly as fast as you can. Make a habit of not passing the buck. The faster everyone gets to the bottom of what went wrong, the faster everything gets back to normal.

This puts you in a position to drive root cause analysis in a way that helps prevent future problems, especially those of a human/organizational nature. I like this piece on a concept called “Five Whys” a lot. It’s simple and elegant. If you’re not using this, you should be using something very much like this.

In the past, I’ve come across a few cases where the position seemed so hard to recruit for, that a company would simply not bother to advertise the vacancy, and instead rely on either filling the position internally, or have someone referred to them. Over the past year, I’ve come to realize that this is simply not the predominant case.

The real problem seems to be that on the operational level, people understand the value a build manager brings to the table, either intuitively or by experience. On the decision-making level however, while that same understanding may also exist, the challenge is quantifying the benefit to justify the expense. Since

1. a build  manager isn’t always perceived as contributing to the deliverable (software), but
2. is billable (contributing to the client invoice),
3. the cost (i.e. salary) is obvious,
4. there is a generally accepted (at least in the agency/consulting world) non-monetary cost of teams working long days and even weekends as a result of those inefficiencies that would typically be mitigated by hiring a build manager (seeing as how non-trivial overtime compensation is rare, even in cases where the client is getting billed for these hours), and
5. the accountability for hiring a build manager (salary) and the cost of not doing so (project inefficiencies, overages, shrinking margins, etc.) is generally with different people (e.g. management and project management respectively),

decision-makers are generally reluctant to explore the fuzzier benefit side of the equation, because all they see clearly is how much money it will cost in salary, which is all they’re likely to have to explain themselves for.

So let me try and summarize:

• The principal job of a build manager is to automate him/herself out of a job. Predictable builds make for predictable results, which makes setting and meeting deadlines a whole lot easier.
• If you can actually count on X amount of work taking Y amount of time to complete, without the process of moving changes through the various test environments to production becoming a problem in and of itself, you can count on your margins, everyone makes money, and everyone gets to go home and have lives.
• Happy employees don’t quit, so you can count on reduced turnover.
• Since you can count on your timelines better than before, you can now be more aggressive in your estimates, making you more competitive when bidding on projects.
• Better value for money makes for happier customers. Happy customers don’t shop for better deals. Now you can count on losing fewer clients, and having to spend less time and money attracting new ones. Your sales and marketing teams can now actually focus on growing your business, rather than fighting to sustain it.

Now obviously, there is more to it than simply hiring someone marketing themselves for the role, or claiming to have the experience and/or expertise. So you will need to understand what you’re actually going to be expecting your build manager to do for you. Give them too little control, and they’ll be limited in what they can do to help. Give them too much responsibility, and they’ll be stretched too thin, and things start slipping through the cracks. In my experience, in small to medium-sized organizations, your build manager needs to have control and ownership over at least:

1. Your version control environment(s), including but not limited to tool administration, branch/merge processes, standards and best practices.
2. Your automated build / continuous integration environment(s).
3. All Test, QA, UAT, etc. environments; this is critically important: if developers are permitted to manipulate these environments unsupervised, you will lose predictability of your process, which will severely limit and ultimately defeat the purpose of having a build manager in the first place. You do not want your build manager to be reduced to a crisis manager.
4. All builds and deployments to all non-development environments; again, this is critically important: the only way to guarantee that the process works, is to enforce it at all times. If the process is too restrictive, it needs to be adjusted, not circumvented.
5. In the case of web development, and if at all possible: setup and configuration of the production environment. All other environments are ultimately derivatives of this, so it’s ideally where the standard is set.

At this point you may think to yourself: that’s an awful lot of responsibility for one person to carry. And you’d be right, it is. Which is why filling the position with a relatively junior developer, or worse, on a rotating basis, simply will not do. It is just a terrible idea on so many levels. You need someone with the obvious skills to do all of the above, and command the respect and deference of the team. You also need a strong enough personality to stand up to, and deal with any conflict or disagreement that is bound to occur in any high-pressure environment. And you need someone who does not already have an established place in the development team’s pecking order, unless that place is firmly at the top.

A build manager is a gatekeeper and, as a natural consequence, a potential bottleneck in the process. This takes a lot of practice, confidence, patience, humility, and ultimately a bond of mutual respect and trust that can only be forged over time. And last but not least, you need to expect some growing pains.

(to be continued)

What’s more interesting to me than his overtly subjective rant thinly disguised as analysis, is the overarching notion that every project has its own, unique needs. The reality is that in the overwhelming majority of cases, it doesn’t. It. Just. Doesn’t. Just because you can do something in a non-standard way, doesn’t mean there’s a need to, or that the benefit of doing something your way outweighs the cost of not following whatever the convention (tool-enforced or otherwise) is.

The author obviously comes from a consulting background (as alluded to here), and consultants generally have the luxury of only being around their handiwork for a limited amount of time, after which the hot mess of long-term maintenance and support – along with all the skeletons you had to cram in the closet to make your go-live date – become someone else’s problem. As a result, the true cost of implementing a custom, proprietary build process is mostly hidden from its creators. What makes sense initially and for the duration of an implementation track, doesn’t necessarily make sense in the long run, once you factor in the learning curve and the cost of supporting and maintaining your fancy homegrown build process.

Maybe, just maybe, conventional build processes work for conventional deliverables. Maybe, just maybe, the conventional wisdom is actually wisdom. I’ve spent enough time fixing other people’s build processes over the past decade or so to know that, generally speaking, when developers are let of the leash long enough to whip up something ‘custom’, the driving force is generally ‘to do something interesting or cool’ before it is actually solving an actual problem. And while the solution may in and of itself be good, the very fact that it’s different from anything else often becomes its own world of trouble.

What I’ve learned more than anything is that building software – like developing it – is its own discrete art form, and one developers tend to suck at. Not for lack of skill, but for lack of focus, and oftentimes a blatant conflict of interest. There is virtue and value in doing things the ‘conventional’ way, whatever that may be. And for all its faults, Maven lets you hit the ground running. You can’t honestly compare it to Rake/Raven/Ant without considering the cost of building, maintaining and supporting a company-standard toolkit based on any of those three, that gives you the functionality you get from Maven out of the box. It’s entirely likely that there are organizations that have legitimately unique needs and requirements, or that operate on a scale where the benefit of a (relatively) custom tool stack outweighs its cost. I will however argue that you, the reader, are far more likely to not work for one.

While Git certainly has its followers (Android being a prominent one), Mercurial offers pretty much the same range of features, in a much more user-friendly package. The latter got a shot in the arm with Google Code announcing their support for it earlier this year.

What I’ve been asking myself is: will this technology work in the enterprise?

There are a number of obvious concerns, which are closely tied to the stated advantages of DSCMs:

1. There is no server / Everyone is their own server. Companies rely on the centralized repository concept to manage backups. The model ensures that anything that is committed to version control, lives in a safe place, and is accessible to all those who need and have appropriate permission. While DSCMs allow for the creation of de facto servers, and processes can be defined to enforce some measure of good order, any such scheme relies almost exclusively on individual cooperation, and tools offer little or no help by way of auditing compliance. In many ways, a DSCM’s greatest strength is its greatest weakness.
2. What doesn’t exist, can’t be secured. Without a centralized server that everyone is forced to read from and write to, how can fine-grained access control (particularly read access) be implemented? The current crop of tools doesn’t cater to this at all: every user essentially has read/write access to anything they can get their hands on. Only when pushing your changes to another user or ’server’ you may encounter write restrictions. This is going to be an instant deal breaker for a lot of organizations. Someone needs to come up with a solution to this problem for DSCMs to gain broad commercial appeal the way Subversion has.
3. Who’s working on what, where and how do I get my hands on it? If development is distributed, how is this then reconciled with a presumably centralized ALM (Application Life cycle Management, i.e. task/issue/defect/project/requirements management) solution? For development to be truly distributed, developers need a distributed ALM solution as well. How this looks in practice or even conceptually, now that’s where someone is going to make a name for themselves.
   ]]> http://www.dutchguybuilding.com/2009/11/03/distributed-version-control-bridging-the-gap/feed/ 1 http://www.dutchguybuilding.com/2008/10/23/accesssvn-100-released/ http://www.dutchguybuilding.com/2008/10/23/accesssvn-100-released/#comments Thu, 23 Oct 2008 15:32:33 +0000 RvO http://www.dutchguybuilding.com/2008/10/23/accesssvn-100-released/ For the past few months I’ve been product-managing a new Subversion management tool called AccessSVN, and today it’s been officially released.

   If you support a large number of SVN users, and you need to manage permissions, this is the only tool on the market for you today. I highly recommend you get yourself a 30-day evaluation license and see for yourself at http://www.polarion.com/products/svn/access_svn.php. When you do, let me know what you think in the comment thread of this post.

   ]]> http://www.dutchguybuilding.com/2008/10/23/accesssvn-100-released/feed/ 0 http://www.dutchguybuilding.com/2008/09/25/the-wasserman-principle-part-i/ http://www.dutchguybuilding.com/2008/09/25/the-wasserman-principle-part-i/#comments Thu, 25 Sep 2008 18:32:26 +0000 RvO http://www.dutchguybuilding.com/2008/09/25/the-wasserman-principle-part-i/ There is this crazy theory I’ve been working on, on and off for probably over a year now. I’m calling it The Wasserman Principle. I got sidetracked for a while, but I’ve been wanting to finish this. In its present form it’s a rambling, unfinished document that needs some work. I’m going to start editing and revising from the top and release hopefully coherent bits and pieces as I go. So here it is, part 1 of TBD.

   With every intended change to software, there is a probability that unintended, unforeseen or unrelated issues will be introduced, triggered or uncovered. These issues need to be addressed as part of the primary work at hand, or deferred. The probability of such issues arising is driven by a number of factors, including, but not necessarily limited to:

   1. Software that undergoes continuous change, such as having new functionality added to its original design, will eventually become more complex and can become disorganized as it grows, losing its original design structure. This is the textbook definition of Software Entropy.
   2. The tendency for a software development organization to be good at some things, and not as good at others. For example: a company may be good at building large applications, but not as good at supporting them (bug fixes, relatively minor enhancements; I’ve worked with and for a few consulting companies over the years that fit this bill), while another company may excel at smaller projects, but struggle with very large ones.

   For the purpose of this theory, entropy (E) is defined as “the probability that intended changes (C) cause something else to break (I).” By keeping track of the actual values for I and C, you can deduce E for your project or organization:

   E = I / C

   With a historical value for E in hand, and for a given number of future changes, issues can be crudely forecasted:

   I = C * E

   So far the math is very simple but abstract, and as a result not very usable in any kind of real-life scenario. I’ll be adding more and more layers of complexity and detail as we get deeper into the subject matter, but this is enough for now.

   ]]> http://www.dutchguybuilding.com/2008/09/25/the-wasserman-principle-part-i/feed/ 0 http://www.dutchguybuilding.com/2008/09/25/i-need-a-hug/ http://www.dutchguybuilding.com/2008/09/25/i-need-a-hug/#comments Thu, 25 Sep 2008 17:30:05 +0000 RvO http://www.dutchguybuilding.com/2008/09/25/i-need-a-hug/
   ]]> http://www.dutchguybuilding.com/2008/09/25/i-need-a-hug/feed/ 0 http://www.dutchguybuilding.com/2008/03/11/wagons-west/ http://www.dutchguybuilding.com/2008/03/11/wagons-west/#comments Tue, 11 Mar 2008 21:01:31 +0000 RvO http://www.dutchguybuilding.com/2008/03/11/wagons-west/ I’ll be at EclipseCon next week as part of the Polarion delegation. For the most part I’ll be behind the scenes, but I should get a chance to open my mouth in the Subversion BOF we’re hosting Monday night.

   

   ]]> http://www.dutchguybuilding.com/2008/03/11/wagons-west/feed/ 0 http://www.dutchguybuilding.com/2008/02/23/subversion-diy-write-through-proxy-lockunlock/ http://www.dutchguybuilding.com/2008/02/23/subversion-diy-write-through-proxy-lockunlock/#comments Sat, 23 Feb 2008 04:02:17 +0000 RvO http://www.dutchguybuilding.com/2008/02/23/subversion-diy-write-through-proxy-lockunlock/ Something I didn’t cover in my original description was how to handle the distribution of pessimistic locking. This is a bit more of an edge-case than some of the others. You need to make sure you understand your options and their side effects.

   There is no clean way to replicate the lock. There are two ways to work around this:

   Propagating calls through the client

   Setup pre-lock and pre-unlock hooks on the master repository, and have them mimic the lock/unlock operations to each mirror. Reason we’re using pre-lock and pre-unlock rather than post-lock and post-unlock is because you need to know which user is running the operation, and that information is not available in the post- hooks.

   You’ll need a separate password file for http://mirror/repo-internal in which you mirror each user, and give them all the same password. That way, you can grab the username in the hook script, and simply hardcode the password. You need real authentication on the internal mirror <Location>, because locking won’t work without it.

   A side-effect of this approach is that the lock tokens on master and mirror(s) will be different. The working copy will have the same token as the master repository, so the lock will appear ’stolen’ to the client, because 'svn st -v' will query the mirror and not proxy through to the master. Other than that, this configuration essentially works.

   Replicate the relevant parts of the repository database

   Setup post-lock and post-unlock hooks on the master repository, and have them rsync the contents of the <repository root>/db/locks directory from master to mirror(s). This ensures the lock token is the same between client, mirror and master. Of course, any untimely breakdown in connectivity between master and mirror(s) in this scenario could lead to unpredictable results and repository corruption. None of this should endanger the integrity of the master repository however, and recovery should be fairly straight-forward.

   ]]> http://www.dutchguybuilding.com/2008/02/23/subversion-diy-write-through-proxy-lockunlock/feed/ 0