TDD – Test-Driven Development – is one of the tools people use to improve their programming. Many teams find that this discipline helps them reduce their defects. Could it make a difference to your team? Explore with this calculator.
What is Improvement?
To call something a process “improvement” suggests we have some way of knowing that things are actually better. You may rely on intuition, but often you can measure something to decide whether it is better.
That measurement could be user satisfaction, team satisfaction, code metrics, defect counts, etc., but businesses like to find ways to put those into a common unit: dollars. [Substitute your currency here.]
This is not to say you can only justify changes with dollars, just that if a change requires a financial investment, you probably want some clue about the payback.
TDD – Test-Driven Development
We’ll use a very simple model for the benefits of doing Test-Driven Development:
+TDD => Defects↘ by x%
Try It
| Assumptions | ||
|---|---|---|
| TDD | ||
| Reduce Defects (%): | Your estimate of TDD’s impact | |
| Team | ||
| Team Size (#): | Number of team members | |
| Burdened Cost ($): | Cost/team-member/month; estimate as salary * 1.3 | |
| Working Days (#): | After holidays, sickness, overhead days, … | |
| Defects | ||
| Defects/Month (#): | Average # of defects fixed | |
| Avg. Days/Defect (#): | Total to find, fix, confirm, deploy, … |
| Calculated Costs | ||
|---|---|---|
| Type | Amount | Formula |
| Monthly Cost of Team | Team-size * Burdened-cost | |
| Cost per Person per Working Day | Burdened-cost / Working-days | |
| Total Defect Cost per Month | Defects/Month * Days/Defect * Cost/Day |
| Potential Savings Per Month Due to TDD | |||||||
|---|---|---|---|---|---|---|---|
| Reduce Defects | Reduce | Defects | Days Spent | $ Spent | % Resources | Days Saved | $ Saved |
The Full Calculator Model
Here’s the full model:
Assumptions
- TDD% = Reduce Defects (%) – Your estimate of the impact of TDD: by what % will defects be reduced?
- TS = Team Size – How many team members do you have?
- BC = Burdened Cost ($) – How much does it cost for the average team member? This is not just salary but also includes overhead for health care, retirement, days off (for whatever reason), equipment, office space, etc. If you don’t know those costs, you can multiply the salary times 1.3 as an estimate.
- WorkDays = Working Days – How many days per month does your team work on average? (There are about 22 weekdays per month on average.) Subtract off the average days for vacation, holidays, sickness, training, and other non-working days.
- Def/Mo = Defects per Month – Your average number of defects fixed per month.
- Day/Def = Average Days per Defect – What is the average effort to do everything associated with fixing that defect: find, reproduce, fix, verify, deploy etc.?
Calculated Costs
- TeamCost/Mo = Avg. Monthly Cost of Team – TS * BC
- Cost/Person-Day = Cost per Person per Working Day – BC / WorkDays
- DefCost/Mo = Total Cost of Defects per Month – Def/Mo * Day/Def * Cost/Person-Day. Note that this only considers the cost of addressing the defect, and doesn’t consider cost to the customers, or other costs to ourselves.
Potential Savings Due to TDD
This table looks at four scenarios, one per row:
- Baseline | 0% – the default situation, assuming no improvement
- …by | 40% – assumes TDD reduces defects by 40%
- …by | 90% – assumes TDD reduces defects by 90%
- Your Estimate | [TDD%] – assumes TDD reduces defects by the % you chose
Why 40% and 90%? It’s hard to estimate the impact of a change in technique. The Nagappan et al. study (from 2008) attempted to compare similar commercial projects using (or not using) TDD, and credited it with a 40-90% decrease in defects. Note however that the teams’ productivity may have been lower.
For each row, there are several columns. The equations below use [field] to refer to a previous column.
- Reduce: the % reduction from TDD (hardcoded, or from your assumption)
- Defects: (1 – [Reduce] / 100) * Def/Mo, the number of defects assuming the “Reduce” %.
- Days Spent: [Defects] * Day/Def, the number of days required to address that number of defects
- $ Spent: [Days Spent] * Cost/Person-Day, the corresponding cost
- % Resources: 100 * [$ Spent] / TeamCost/Mo, the % of the team’s cost consumed by defect fixing. (This one can be a shocker.)
- Days Saved: [Baseline.Days Spent] – [Days Spent], how many fewer days are spent using TDD, compared to the baseline row.
- $ Saved: [Baseline.$ Spent] – [$ Spent], how many fewer dollars are spent using TDD, compared to the baseline row.
Limitations and Generalizations
Notice some limitations of this model. It:
- Assumes a very simple model of impact
- Ignores the cost of learning to do TDD
- Ignores how long from introducing TDD until the benefits are felt
- Assumes productivity remains stable
Double-check the reasoning and the math before writing any big checks! Test whether the assumptions and model apply to you before relying on it too deeply.
This spreadsheet is labeled “TDD”, “defects”, etc. But you could assess other changes by changing the labels, provided the simple model is adequate.
Further Reading
- Resources on Test-Driven Development (TDD)
- More articles on TDD at xp123.com
Acknowledgements
A conversation with Chris Pitts a couple years ago got me thinking about this model (though any mistakes are mine).