[Originally appeared in Methods and Tools, Winter 2002 edition.]
Software is hard. It’s hard to find out what’s needed. The real requirements are hard to discover; plus, they change over time, and they change as a result of the software being created. But even once we know what’s wanted, software is still hard. It’s hard to master many details, and it’s hard to merge together the results of a team’s work. Extreme Programming—XP—is a software development method designed to help teams create software more effectively. XP uses “simple rules” as a starting point for a team’s process. XP’s claim is that if we:
- Put the whole team in a room together,
- Force feedback through constant planning, integration, and release, and
- adopt a test-driven approach to programming
then the team can be highly responsive and productive.
We’ll look at XP as a series of nested conversations. You’ll see a pattern repeated at each level: explore, plan, define a test, do the work, verify the result.
| Nested Conversations in XP | |
|---|---|
| Month scale | Release planning, iterations, release to users |
| Week scale | Iteration planning, daily work, release to customer |
| Day scale | Standup meeting, paired programming, release to development team |
| Hour/Minute scale | Test, code, refactor |
Three Voices
Let’s consider three situations where you might develop a program.
First, suppose you have a problem, and the resources, time, ability, and desire to write a program (and that’s the easiest thing to do). Then you may write the program to meet your needs. You may feel no need to separate roles.
Next, suppose you are a business owner and you have the resources and a need, but no ability or desire to write a program. You might hire a programmer to do it for you. It makes sense for you to distinguish your role from the programmer’s.
Finally, imagine that you have the vision and need for a program, but you’re not able to write it yourself, and you don’t have the money or resources to hire a programmer. You might convince someone else to provide the money to hire a programmer. The person putting up the money will want to be involved (to be sure the money isn’t wasted).
XP views teams as similar to this final scenario. An XP team has three sub-teams:
- Customer: Someone with needs and vision for a solution
- Programmer: Someone with the technical skill to create a solution
- Manager: Someone who provides an environment and resources for a customer and developer to work together.
The interaction between sub-teams is treated as a conversation between these roles. Underneath, each role is composed of several people, each with their own conflicts and compromises, but when they speak to the other roles, they speak with one voice. Here, we see one of the ways XP simplifies reality. In reality, people have many skills: testing, analysis, programming, etc. But for the sake of the process, we act “as if” reality were simpler.
| Key Points – Three Voices |
|---|
|
The Context
Just as a choir needs rehearsal space and an accompanist, an XP team needs resources: workspace, computers, money, software, people, and so on. Management provides this context. They take responsibility for hiring, firing, space, tools, and so on. They control the flow of their investment; they may increase or decrease it.
XP asks for a special environment: a place where the whole team can sit together. Who is the whole team? It includes all the roles, but especially Customer and Programmers. Sit together means “really” together: in the same room (not in cubicles in the same building), where people can see each other. (XP is not saying a good software team can’t be split across locations; it’s saying that a team can be more productive if it sits together.)
People who are near each other talk to each other more. XP teams value communication and feedback, and collocation feeds both.
| Key Points – The Context |
|---|
| The whole team sits together. |
Some Vocabulary
A release is the delivery of software, ready for use. Ideally, and usually, this is “use” by real end users, so the team gets feedback. A release is typically developed and delivered in one to three months, but some teams deliver every week or even more frequently.
An iteration is a time period during which the team develops software. Iterations have a fixed length, usually one or two weeks. Iterations are time-boxed: if a feature won’t be done as planned, the scope of the iteration is adjusted—the iteration is not changed in length.
The Month-Scale Conversation
The whole project looks like this:
Release planning | Iteration | … | Iteration | Release
The conversation begins with an iteration or two of planning, and ends with a release to the end users.
To decide what will go into a release, the team will create a release plan. This plan shows, for each iteration, what features the system should have.
To create this plan, the customer will create stories–brief description of features, written on index cards. Here is a sample story card:
| Create Account Enter desired username, password, & email address. Re-try if already in use. |
The customer describes the story, but not every detail is necessary at this time. Rather, the conversation needs enough detail that the programmers are comfortable giving a ballpark estimate.
There will be a lot of conversation, but it won’t be all talk for the week or two. The programmers have several jobs during this exploratory time:
- estimate how long each story will take to implement
- make quick experiments, known as spikes, to better inform the estimates programmers will assign
- set up the development and test environment
- create a super-simple skeleton version of the application, to demonstrate the pieces working together. (For example, a web-based course registration system might have one web page with a simple form of one field, sent to a back end that looks up one thing in the database.)
Most XP teams use a simple relative scale for the stories: 1, 2, 3, or “too big.” (These numbers are often called story points.) It’s helpful to have a rule of thumb as a starting point; you can use “one point = the team for a day” or “one point = a programmer for a week.” But mostly, the stories will be compared against each other: “Is this one about as big as that one? Do we understand how to do it?” If a story is too big, the customer (not the programmers!) will divide it into two or more smaller stories until it can be estimated.
Using relative estimates is a little unusual, but it does have some benefits. It lets teams estimate stories by their “inherent” complexity, independently of the details of who will do exactly what part of the work. Furthermore, teams are often better able to start by saying, “these are about the same size” than to estimate a story’s absolute cost. The team’s ability to implement stories may change over time as a team learns and changes, but the relative costs of many stories will tend to stay the same.
The customer will write stories until they feel they’ve explored the possibilities enough to make a plan, and the programmers have estimated each story.
Given the estimated costs, the customer ranks the stories from most important to least. The programmers give an estimate of velocity: the number of story points they believe the team can implement per iteration.
The customer will arrange the stories into columns, with “velocity” points per column. The number of columns is determined by the number of weeks the customer (or management) wants to go until release. If all the stories don’t fit, the customer can either defer them to a new release, or adjust the planned release date. (For most customers, it’s better to drop features rather than slip a date.)
Here is a sample release plan, for a team with a velocity of four points/iteration:
| Release Plan |
|---|
| Iteration 1 List courses (1) Enroll in course (2) Summary report (1) |
| Iteration 2 Create account (2) Login (1) Show schedule (1) |
| Iteration 3 Drop course (1) Manage courses (3) |
| Iteration 4 Fees (2) Logout (1) 1-sec. response (1) |
This is a plan, not a commitment. It will change over time.
Until the release, the team engages in iterations. We’ll look at the iteration-level conversations next.
Key Points – Month-Scale Conversation
|
Week-Scale Conversations: Iterations
A release plan is a high-level plan; it’s not detailed enough for a team to work from directly. So, an iteration begins by creating an iteration plan. This is similar to a release plan, but at a finer level of detail.
The team first asks, “How many story points did we complete in the last iteration?” The team will plan for that many points for this iteration. This rule is known as Yesterday’s Weather, on the theory that yesterday’s weather is a pretty good predictor for today’s. (For the first iteration, the team can use the velocity estimate from the release plan.)
The customer selects the stories for the iteration. The customer needn’t select stories in the order used for the release plan; they can pick whichever stories have the most value given what they now know. They might even have new stories for the team to estimate.
The team then brainstorms tasks, to form an iteration plan. For each chosen story, the team lists the tasks that will implement it. One of the tasks should be “run customer test” (or the equivalent), as the customer’s criteria determine when the story is done. Here is a sample iteration plan:
| Iteration 2 Plan Create Account (2)
Login (1)
Show Schedule (1)
|
Individuals then sign up for tasks. Some teams assign all tasks at once, others sign up as the iteration progresses. In either case, you should be able to look at the chart and know who’s doing what.
Note that “Run customer test” is a task for each story. One of the jobs of the customer is to specify a test for each story. Ideally, the tests are ready even before iteration planning, but they need to be done before a “completed” story can be counted. Ron Jeffries describes requirements in XP as having three parts, “CCC” –Cards, Conversations, and Confirmation. Customer tests provide the confirmation. (Recall that “customer” may be a team including professional testers. The customer need not implement the test, but should specify and own it.)
Once we have a plan, the conversation shifts down to daily activities. At the end of the iteration, the team will deliver the application to the customer for final assessment of the iteration’s work.
Key Points – Week-Scale Conversation
|
Day-Scale Conversations: Daily Activities
The team begins its day with a stand-up meeting. (This practice was adapted from the Scrum process.) The team stands in a circle, and each person takes a minute to tell what they did yesterday, what they plan to do today, and what’s in their way. (If they need help, they can ask for a followup meeting.)
About halfway or two-thirds through the iteration, the team should do what Ward Cunningham calls a “Sanity Check”: “Are we on track? Will we finish the stories we planned? Do we need more stories?” If the team is behind, they may need to re-negotiate the iteration plan with the customer: defer a story, choose a simpler one, etc. (These adjustments – dropping or adding stories – are what allow velocity to change from iteration to iteration.)
After the standup meeting, the team will break into pairs – groups of two. XP specifies that production code will be implemented by pair programming, for immediate design and code review. (See Pair Programming Illuminated.)
The pair selects a task, and works on it together. Periodically, they integrate their work into the main body of code, or pick up what others have integrated. After a couple hours, the pairs may swap around.
At the end of the day, the pair will integrate one last time. If they’ve completed a task, they’ll mark it off. (Very rarely, they may decide they’re on a wrong track, and abandon the last bit of work.)
Several things enable such flexible teamwork:
- The team has daily standup meetings and sits together, so everybody has some idea of what everybody else is up to.
- The team owns the code jointly; any pair can change any code it needs to.
- Each pair integrates its work into the mainline many times per day.
- The mainline is kept working: no code is checked in that causes a regression in the tests.
Key Points: Day-Scale Conversation
|
Hour- and Minute-Scale Conversations: Programming
The programmers talk to each other, and the customer, as they work on their task.
There is also a “conversation” with the code. XP uses a development style known as test-driven development. You’ll hear many different words and phrases associated with this technique: simple design, test-first programming, refactoring, green-bar/red-bar.
To create a new feature, the programmer writes a new, small, “programmer” test for part of the feature. The test fails, of course, since the feature doesn’t exist yet. So the programmer implements the feature in a simple way to make the test pass. Then they refactor (systematically improve the design) to remove any duplication and leave the code communicating as well as it can. The whole cycle repeats until the feature is completely added. Each trip through the cycle takes a few minutes.
At the end of a session, the program will have a new feature, and an automated set of programmer tests that demonstrate it. The code and tests go into the mainline, and the team will keep both the feature and the tests working while other code is added.
Programmers have found that this style yields programs with decoupled and encapsulated designs, and the code is known to be testable (as it has been demonstrated).
Key Points: Hour/Minute-Scale Conversation
|
Beyond the Mechanics
The description above represents a starting point for a team’s process. Real teams will evolve their process over time.
Teams have found that there is a synergy in the XP practices. They support each other, so you can’t just pick and choose without finding some new practices to balance the missing ones.
Many teams have found that adding a regular retrospective is helpful: it builds in time to reflect on how things are going and how they can be improved.
Conceptual Frameworks
We’ve looked at XP from the “mechanical” side; now we’ll consider its underpinnings from some other perspectives:
- Values and practices
- Agile methods
- Self-organization at the team level
- Empirical vs. defined processes
- Emergence at the code level
- Lean Manufacturing
Values and Practices
The first XP book, Extreme Programming Explained (by Kent Beck), introduced a framework of “values” and “practices” for describing XP. Values are more fundamental; practices are activities or skills that are compatible with the values, and form a starting configuration of team skills.
The values:
- Communication
- Feedback
- Simplicity
- Courage
The practices:
- On-site customer
- Planning Game
- Metaphor
- Short Releases
- Testing
- Continuous Integration
- Collective Ownership
- Forty-Hour Week
- Pair Programming
- Simple Design
- Refactoring
- Coding Standards
(Most of these practices were worked into the description of the mechanics described in the first part of this paper. There are other lists of practices that use different words to explain the same themes.)
Agile Methods
Extreme Programming is an example of what are known as agile methods. Some others include:
- Scrum (www.controlchaos.com): This is probably the most philosophically compatible with XP. Scrum uses one-month iterations, and its own approach to planning. It allows for large projects via a “Scrum of Scrums.”
- Crystal Clear (www.crystalmethodologies.org): “Management by milestones and risk lists.” Crystal Clear is the simplest in a family of methods.
- FDD (www.featuredrivendevelopment.com): Plan, design, and build by feature, in a model-driven approach supported by a chief programmer.
- DSDM (www.dsdm.org): Model and implement through time-bound iterations. (DSDM is an outgrowth of earlier RAD approaches.)
See www.agilemanifesto.org for a manifesto and principles statement from a number of leaders in agile methods, and www.agilealliance.com for the home of the Agile Alliance.
Self-Organization of the Team
Among agile methods, XP and Scrum stand out as relying on a team to organize itself. This flows from the team taking on responsibilities. It’s also due to the lack of built-in role specialization. XP teams value specialized skills; but they don’t pigeonhole people into having only one aspect. (Database programmers who want to learn some GUI programming can pair with someone who has more experience.)
The team takes responsibility: the team accepts stories, and the team finds a way to do them. In most XP teams, individuals accept tasks. Even so, they’re understood to have the full support of the team. If they ever need help, they ask, and it will be given.
The physical environment encourages self-organization too. When people sit together and eat together, they build bonds and realize, “We’re in this together,” and “What affects you affects me.”
Defined and Empirical Processes
(Scrum brings this vocabulary into play as well.) Consider making cookies. You have a recipe, and you follow it. If you make another batch, with the same ingredients, in the same proportion, in the same oven, you expect to get the same result. This is an example of a defined process.
Consider instead the process of creating a cookie recipe. The value comes from its originality. You might try a number of variations, to get just the right result. Iteration is inherently part of the process; this is an empirical process. (Reinertsen, Managing the Design Factory, suggests the cooking example.)
Software development tends to be an empirical process: the goal is not to get the same result a team got earlier, but to create something new. Experimentation is a critical part of this, not a failure.
In spite of the concrete description in the first half of this article, XP is in the “empirical” camp. It accepts that there will be experimentation on all levels, including experiments about the process itself.
Emergence at the Code Level
One of the unexpected aspects of XP is its flipping around the development cycle from “analyze- design-code-test” to “analyze-test-code-design” (Ralph Johnson). One way to design is to speculate on the full design that will be needed. Another approach is to intertwine design and development. XP follows the latter approach: build a little something, then evolve and generalize the code to reflect the design.
Martin Fowler’s book Refactoring catalogs “code smells” (indicators of design problems) and “refactorings” (safe transformations that can address problems). These provide (usually) local improvements to code.
The team also looks for more global improvements. Pair swapping and shared ownership mean that people will be exposed to more areas of the code, so able to spot similarities among disparate sections. The team’s search for a metaphor (shared understanding of the system) can help this too.
Why is this emergence? Because simple rules (smells and transformations) lead to something perhaps unexpected: globally good design.
Lean Manufacturing
The automobile industry has moved from assembly lines to lean manufacturing. Traditional assembly lines “push product” as fast as possible; inventory is regarded as an asset. In lean approaches, a product is “pulled” from the system, and inventory is regarded as a source of waste.
XP’s approach to planning and implementation strives for “just in time” work.
- At first, stories described with just enough detail to allow an estimate.
- For each iteration, just enough stories are expanded with tests and details so the stories can be broken into tasks.
- For the current task, then the pair will write a test and implement just enough code to make the test pass.
- For the resulting code, the pair will refactor to reflect the design as it’s now understood.
Suppose a team is doing 100 stories, at the rate of ten per week. This might be the schedule for an XP team:
- Week 1: 100 stories described and estimated
- Week 2: 10 stories get customer tests; those same 10 stories get unit-tested, coded, and refactored.
- …
- Week 11: The last 10 stories get customer tests, unit-tested, coded, and refactored.
Because the team is completes the highest-value stories first, the earliest iterations are the most valuable.
Compare this to a strict waterfall:
- 100 stories get analyzed
- 100 stories get designed
- 100 stories get coded and unit-tested
- 100 stories get tested
At some level, there’s the same amount of total work (though my bet would be on the first team). But look at it from a flow perspective: we don’t see any results until stories come out of testing.
Think of a story as inventory. When it has been analyzed, designed, and coded, but not tested or deployed, it has a substantial investment at risk. The XP pipeline lowers the risk: a story gets everything done at once, and spends less time at risk.
Conclusion
XP challenges traditional software development processes in several ways: everything from how a team is structured to how code is implemented comes in for scrutiny. The XP practices represent an effective way to help a team learn what software is needed and develop that software, while respecting and valuing each person on the team.
Further Reading
- The XP series from Addison-Wesley: Extreme Programming Explained (Kent Beck) is the first in the series; Extreme Programming Explored is my contribution; Testing Extreme Programming (Lisa Crispin and Tip House) is the latest addition.
- The Agile Software series, also from Addison-Wesley: Agile Software Development (Alistair Cockburn) is a good starting point.
- Managing the Design Factory: The Product Developer’s Toolkit by Donald Reinertsen
- Pair Programming Illuminated, by Laurie Williams and Robert Kessler
- XP web sites: extremeprogramming.org, xp123.com, and xprogramming.com
- “XP on One Page” is a mini-poster describing XP