An acceptance test is a test that the user defines, to tell whether the system as a whole works the way the user expects. Ideally, the acceptance tests are defined before the code that implements the feature.
Acceptance tests are run frequently in an XP project, usually daily or more often, and certainly every week. The tests give a picture of how much of the desired functionality will work. Because the tests are run so often, there is a payoff if they can be automated.
Progress on acceptance tests is one of the crucial metrics that XP projects often collect.
There are several mechanisms that can be used to run acceptance tests:
- Manual Test
- GUI testing tools
- Code
- Script
- Spreadsheet
- Template
Manual Test
The simplest mechanism for running a test is to just make a plan on paper, and then do the steps manually.
GUI Testing Tools
There are commercial tools that let you run a system, recording your activities and the system’s responses. You run it once to establish a “golden” copy, and then automatically after that to verify that things haven’t changed.
This might be useful for some parts of testing, but it’s not as good as you might think.
- If the interface changes, the tests are invalidated. (The tools’ exact approach determines how significant an interface change will trigger this problem.)
- The interface is often (usually?) not the core of the problem being solved. Thus it might be the case that interface is done toward the end of the project. All of the functionality should be tested along the way.
Code
In the code approach, you get a programmer to write the code that will run the tests you specify. This code is usually straightforward. Some teams use frameworks such as JUnit to manage and run the acceptance test code.
Script
“Script” is a simplified form of code. Programming languages typically have several ways to repeat actions, to do something if a condition is or isn’t true, and so on, but most test code doesn’t need these more complicated parts of programming.
So, programmers might create a scripting language. Tests are usually stylized, simple code, and the programmers will provide a way to run them. The language might be a subset of the one the programmers are using (e.g., Java), or a more “scripting” language like Perl, Ruby, or Python.
Example:
doc("Extreme Programming Explained", "Kent Beck", 1999);
doc(“Planning Extreme Programming”, “Kent Beck and Martin Fowler”, 2000); expect(“Extreme and Programming”, 2); expect(“Explored”, 1); expect(“Fish”, 0); expect(“Explored or Planning”, 2);
Downsides: Script languages still require a lot of attention to details, and may have unwanted complications. Even in this simple example, there are several issues that the test writer has to be careful about:
- Don’t forget the semicolon at the end of each line.
- How do you put quotes inside quotes?
- Numbers vs. strings
Benefits: It’s not all bad news:
- It’s easy to get started (by adapting an existing language)
- Gives the user direct ownership of the tests
- The solution is flexible.
Providing a scripting language is often the easiest way to get a user writing tests.
Spreadsheet
When you get to their essence, many tests can be represented in a spreadsheet. This gives the user a well-understood way to create, edit, and manage tests.
Example:
| Title | Author | Year |
|---|---|---|
| Extreme Programming Explored | Kent Beck | 1999 |
| Planning Extreme Programming | Kent Beck and Martin Fowler | 2000 |
| Query | Expected | Comment |
|---|---|---|
| Extreme and Programming | 2 | AND found |
| Explored | 1 | No booleans |
| Fish | 0 | Term not present |
| Explored or Planning | 2 | OR found |
Spreadsheet formats are not hard for a program to handle. Spreadsheets almost always have a way to save files in a format where commas, tabs, or “|” bars separate the fields. Then the program just reads one row per line and does what is needed. Different types of tests might use different programs, but still work with the spreadsheet approach.
Template
As you grow a bunch of different tests, it becomes more of a pain to write a new small program for each test type. You can use a “template” approach that combines scripting and spreadsheets.
The user writes the script that would test one row, and then “templatizes” it by replacing data values with a generic name. Then one program can create the actual script by merging the template and the spreadsheet.
Example:
*{
test($Comment);
expect($Query, #Expected);
}*
In this made-up example template, the part contained in the “*{ }*” brackets is repeated once per spreadsheet row.
Conclusion
There are several options for creating and running acceptance tests. Each team will have to decide which option (or combination) will work best for it.
Resources and Related Articles
[May 15, 2001.]