Acceptance Test Mechanisms

XPlorations

XP123 → XPlorations → Acceptance Test Mechanisms (May 2001)

Acceptance Test Mechanisms

May, 2001

An acceptance test is a test that the user defines, to tell whether the system as a whole works the way the user expects. Ideally, the acceptance tests are defined before the code that implements the feature.

Acceptance tests are run frequently in an XP project, usually daily or more often, and certainly every week. The tests give a picture of how much of the desired functionality will work. Because the tests are run so often, there is a payoff if they can be automated.

Progress on acceptance tests is one of the crucial metrics that XP projects often collect.

There are several mechanisms that can be used to run acceptance tests:

Manual Test
GUI testing tools
Code
Script
Spreadsheet
Template

Manual Test

The simplest mechanism for running a test is to just make a plan on paper, and then do the steps manually.

GUI Testing Tools

There are commercial tools that let you run a system, recording your activities and the system's responses. You run it once to establish a "golden" copy, and then automatically after that to verify that things haven't changed.

This might be useful for some parts of testing, but it's not as good as you might think.

If the interface changes, the tests are invalidated. (The tools' exact approach determines how significant an interface change will trigger this problem.)
The interface is often (usually?) not the core of the problem being solved. Thus it might be the case that interface is done toward the end of the project. All of the functionality should be tested along the way.

Code

In the code approach, you get a programmer to write the code that will run the tests you specify. This code is usually straightforward. Some teams use frameworks such as JUnit to manage and run the acceptance test code.

Script

"Script" is a simplified form of code. Programming languages typically have several ways to repeat actions, to do something if a condition is or isn't true, and so on, but most test code doesn't need these more complicated parts of programming.

So, programmers might create a scripting language. Tests are usually stylized, simple code, and the programmers will provide a way to run them. The language might be a subset of the one the programmers are using (e.g., Java), or a more "scripting" language like Perl, Ruby, or Python.

Example:

    doc("Extreme Programming Explained", "Kent Beck", 1999);
    doc("Planning Extreme Programming", "Kent Beck and Martin Fowler", 2000);

    expect("Extreme and Programming", 2);
    expect("Explored", 1);
    expect("Fish", 0);
    expect("Explored or Planning", 2);

Downsides: Script languages still require a lot of attention to details, and may have unwanted complications. Even in this simple example, there are several issues that the test writer has to be careful about:

Don't forget the semicolon at the end of each line.
How do you put quotes inside quotes?
Numbers vs. strings

Benefits: It's not all bad news:

It's easy to get started (by adapting an existing language)
Gives the user direct ownership of the tests
The solution is flexible.

Providing a scripting language is often the easiest way to get a user writing tests.

Spreadsheet

When you get to their essence, many tests can be represented in a spreadsheet. This gives the user a well-understood way to create, edit, and manage tests.

Example:

Title Author Year

Extreme Programming Explored Kent Beck 1999

Planning Extreme Programming Kent Beck and Martin Fowler 2000

Query Expected Comment

Extreme and Programming 2 AND found

Explored 1 No booleans

Fish 0 Term not present

Explored or Planning 2 OR found

Spreadsheet formats are not hard for a program to handle. Spreadsheets almost always have a way to save files in a format where commas, tabs, or "|" bars separate the fields. Then the program just reads one row per line and does what is needed. Different types of tests might use different programs, but still work with the spreadsheet approach.

Template

As you grow a bunch of different tests, it becomes more of a pain to write a new small program for each test type. You can use a "template" approach that combines scripting and spreadsheets.

The user writes the script that would test one row, and then "templatizes" it by replacing data values with a generic name. Then one program can create the actual script by merging the template and the spreadsheet.

Example:

    *{
        test($Comment);
        expect($Query, #Expected);
    }*

In this made-up example template, the part contained in the "*{ }*" brackets is repeated once per spreadsheet row.

Conclusion

There are several options for creating and running acceptance tests. Each team will have to decide which option (or combination) will work best for it.

Resources and Related Articles

[May 15, 2001.]