| What mechanisms can a team use to prevent integration problems? |
This article is available in the articles section at https://www.informit.com as part of a feature on agile methods. Navigate to Home > Articles > Software Engineering > Agile Computing. (Sorry, I can’t give a direct link as their URLs use session IDs.)
Introduction
Continuous integration is one of XP’s important team practices. Rather than weekly or daily builds, XP teams strive to integrate the system several times per day. Such frequent integration has several benefits:
· Integration is much easier because so little has changed since the last integration.
· The team learns more quickly because unexpected interactions are rooted out early on while the team can still change its approach.
· Problematic code is more likely to be fixed because more eyes see it sooner.
· Duplication is easier to eliminate because it’s visible sooner, when it’s easier to fix.
Integration in XP has another advantage that is critically different compared to some other approaches: The team agrees that the system should always be in a working state before and after an integration. That is, integrators are not allowed to leave the system worse than they found it.
In this article, we’ll look at several possible approaches to continuous integration with an eye toward finding an effective approach.
Method 1: Shared Directory
Keep all files in one directory (tree). Anybody can edit the files whenever they need to. The mainline of code–the copy of the code that everybody agrees is the “right” copy from which to develop–consists of whatever is in that directory.
While this may be the simplest method, it’s not the simplest method that will work. (In spite of its faults, I’ve met more than one group using this approach.) This method has some inherent problems:
· Two people editing the same file at the same time can interfere with each other; changes can be lost. The situation can happen in two ways (see Figures 1 and 2). If two people are editing a file, one or the other will save first, without awareness of the other’s changes. Some editors can detect this situation and warn about it but this isn’t always enough.
Figure 1. An overlapping change.
Figure 2. Another overlapping change.
· Without coordination, the mainline may never be in a “good” state. Suppose one pair finishes their changes and starts on their next task. But the other pair is still in the middle of their changes. You may have to go back an arbitrary amount of time to find a consistent version. (Some groups use the idea of a “code freeze” as a time when changes are forced to be integrated, with no new work started.)
· For configuration management purposes, you can take a snapshot and back up the directory at any time, but you may never have a known-working system.
· There’s no concept of a transaction for changes. A transaction is a set of changes that either all succeed or all fail, as a group. Transactions are useful because developers don’t always know whether their approach will succeed. If it fails, you want to avoid making it part of the mainline.
Method 2: Shared Directory Plus Lock
In this method, you use a shared directory as before but add a lock so people aren’t editing at the same time. (A lock ensures exclusive access to a resource; when someone has the lock, nobody else is allowed to use the resource.)
The lock may be virtual (such as renaming a file), or it can be a physical object (such as a stuffed animal held while the code is locked).
Each pair uses this approach:
1. Wait until the lock is free.
2. Lock.
3. Make changes.
4. Back up the system.
5. Unlock.
This strategy has these effects:
· Pairs no longer interfere with each other.
· The team no longer works in parallel (trading speed for safety).
· If all changes are successful, the mainline is in a good state after each release of the lock.
· Changes can be transactional; if a set of changes is unsatisfactory, the pair can restore a previous version (abandoning the unwanted changes).
Method 3: Mainline Plus Sandboxes
To allow the team to work in parallel, we can provide them with a copy of the system in a separate work area known as a sandbox. Changes made in the sandbox have no impact on the mainline.
Now making changes is a multiple-step process:
1. Copy:
a. Lock.
b. Copy mainline to empty sandbox.
c. Unlock.
2. Change:
a. Make changes in sandbox.
3. Integrate:
a. Lock.
b. Integrate sandbox into mainline.
c. If integration fails, restore the prior version.
d. Back up the system.
e. Unlock.
The process looks like Figure 3.
Figure 3. Integration using sandboxes.
One aspect of this process is tricky: “Integrate sandbox into mainline.” You can’t just copy the sandbox in, or you would lose changes made by others.
Integration is easy in one case: If the file is identical in both the mainline and the sandbox, you don’t do anything.
Three cases are more interesting:
· The file is in the sandbox but not the mainline.
· The file is in the mainline but not the sandbox.
· The file is in both mainline and sandbox, but differs.
In each of these cases, the integrator must make a decision about which files–or which parts of files–should be used. (More sophisticated schemes and better tools can help with the process.)
Successful integration means that all tests must still pass when you’re done integrating. If you’re unable to integrate successfully, you can try later, or clear out your sandbox and start over.
This approach has these consequences:
· Pairs can work in parallel.
· Integration is more difficult than it was in the previous scheme.
· If integration fails, the mainline can be restored from its last backup.
Method 4: Mainline Plus Sandboxes Plus Synchronization
Some big changes may require several integrations before the change is complete; this can make our integration tricky. You can address this potential problem by adding a synchronization operation that brings others’ changes into your sandbox.
Synchronization can be mixed in while you’re doing your changes:
1. Synchronize:
a. Lock.
b. Integrate mainline into sandbox.
c. Unlock.
Some source-control systems go further, providing a “preview” that identifies the files that would require integration, without actually bringing the changed files to the sandbox.
With this refinement, you get the following advantages:
· Programmers can still work in parallel.
· Integration is easier.
· Integration is still transactional.
A Few More Observations
· Earlier we talked about configuration management as taking a “snapshot” or backing up the system, but source-control systems optimize this facet in such a way that they don’t need to archive unchanged files. This makes the system faster, but doesn’t change the point: You want to be able to restore the system to a previous state.
· Some configuration systems provide a branch for each sandbox (and most allow them for other purposes as well), so mainline is really in contrast to the branches. Some systems even capture each edit made in the sandbox, making it very easy to do things like undo an unsuccessful refactoring.
· How often does integration occur? I encourage each pair to integrate at least twice a day, and preferably every hour or so.
· Be sure to “go home clean”; don’t leave code checked out (and not integrated) overnight. If a change is bigger than a day’s work, take it as a sign that you should try to find a simpler way to tackle that change.
· Some teams use an integration machine: a separate machine used only for integration (and not for development). At one level, this doesn’t seem like it should make a difference, but it has some advantages:
o Technically, it provides a relatively clean environment; if files weren’t copied from the sandbox to the mainline, the integration won’t succeed. (In most environments, this means that it wasn’t added to source control.)
o Psychologically, the act of moving to a new machine provides a mini-break and a sense of closure.
o Socially, seeing people move to the integration machine several times when you haven’t integrated in a while serves as a mini-jolt to remind you to integrate.
Conclusion
We’ve taken a tour of possible integration approaches. Other schemes can certainly work, but our final scheme provided these benefits:
- People don’t silently step on each other’s work.
- Integration is transactional; new changes are either fully included or fully excluded.
- The system moves from a known-good state to a known-good state.
- The approach is compatible with source control.
[Written February, 2002.]