| Fit's standard interpretation tells us how well a program does against a set of test cases. We can design new semantics for reporters (that give us interesting information) and for rewriters (that make interesting transformations of our tests). |
The Standard Interpretation
A Fit test looks like a table (or a series of tables) in an HTML document:
| Calculator | |||
| x | y | plus() | times() |
| 2 | 1 | 3 | 2 |
| 0 | 3 | 3 | 0 |
| 2 | -1 | 1 | -2 |
| fit.ActionFixture | ||
| start | ScientificCalculator | |
| enter | value | 2 |
| press | plus | |
| enter | value | 1 |
| press | equals | |
| check | value | 3 |
But what does a Fit test mean?
This is almost a trick question. One answer is: whatever people agree it means. Another answer is: whatever the Fit framework, the fixtures, and the application make it mean. These two things don't always line up, unfortunately.
Even when we think we're talking about the same interpretation, there can be differences in what a test means. One version of the Calculator fixture could work with the business objects; another could work at a presentation layer; another could work by manipulating the user interface. These fixtures might find different types of defects; they might be easier or harder to write; and so on.
These all generally assume a standard interpretation: use Fit to run the fixture on the table. The good side of this is that it gives us something we care about: an answer to the question, "How well does our program work?"
I'd like to move to a different question:
What can we know about our tests if we don't understand the fixtures?
Alternative Semantics
A different interpretation of the directory containing my test above might yield this table as a result:
| Index | ||
| Fixture | File | Table Number |
| Calculator | MyTest.html |
1 |
| Calculator | OtherTest.html | 3 |
| fit.ActionFixture | MyTest.html | 2 |
This is a cross-reference chart, showing where every fixture is used. I can get this information with no reference to the fixture implementation at all: it's just an index of the first cell of each table.
We can find out other interesting things with just a little knowledge of fixtures. For example, if we know that "Calculator" above is a ColumnFixture, then we know that the first row consists of input and outputs. Even without knowing anything about how the fixture is implemented, we can create another interesting table:
| Vocabulary | ||
| Fixture | Field | Value |
| Calculator | plus() | 1 |
| Calculator | plus() | 3 |
| Calculator | times() | -2 |
| Calculator | times() | 0 |
| Calculator | times() | 2 |
| Calculator | x |
0 |
| Calculator | x |
2 |
| Calculator | y | -1 |
| Calculator | y | 1 |
| Calculator | y | 3 |
This table gives you a picture of the input and output domains. A good tester could look at this and notice all kinds of things that weren't tested:
- no large or small numbers (in either the input or output)
- no non-numbers
- no sums that resulted in 0
- no sums that were negative
- no sums that were even
- only even numbers for x, odd numbers for y
- minus(), divide()
- etc.
We could create a tool that would give us a domain analysis of our test data:
| Test Data for Calculator | |||||||
| Field | Max Neg | Neg | Zero | Pos | Even | Odd | Max Pos |
| plus() | no | no | no | yes | no | yes | no |
| times() | no | yes | yes | yes | yes | no | no |
| x | no | no | yes | yes | yes | no | no |
| y | no | yes | no | yes | no | yes | no |
Reporters and Rewriters
Two types of interpretations stand out for me:
Reporters – tell you something about a set of tests
Rewriters – change a set of tests
The Index and Vocabulary examples above are reporters. The standard Fit interpretation is close to being a rewriter: it produces a version of the input file, modified to show the results of the tests. (The only thing that keeps it from being a true rewriter is that it leaves the original file in place, so you can run it again.)
Here are some more ideas for useful tools:
- Count test cases- A reporter telling the total number of test cases for a fixture
- Count fixture – A reporter telling how many times a fixture occurs
- Operators – A reporter telling the column names in RowFixtures and ColumnFixtures (or the second column of ActionFixtures)
- Operands – A reporter telling the data values used (like "Vocabulary" above)
- Column changer – A rewriter that can rename, delete, or insert a column
- Cell rewriter – A rewriter that can change cell values (for a specific fixture in a specific column or row)
(I've created a simple version of the Index example above using the AllFiles fixture; the others are speculation.)
Reflection is OK
There are useful semantics that don't try to interpret a fixture, but it can help to "peek" a little. For example, knowing that something is a ColumnFixture tells us that it's likely that the row after the fixture name consists of input and output fields. We can use this information fruitfully. The Vocabulary example above made use of this knowledge.
Furthermore, there is nothing wrong with getting help. If someone had a new type of fixture that subclassed Fixture, but still had ColumnFixture-like semantics, they could provide a helper analysis class that would let us know this.
The goal is not to avoid using fixture-aware code, it's just to avoid the quagmire of trying to interpret another program.
Call to Action
We've had a few years to work with Fit. People are creating test suites large enough to be interesting, and large enough that they need help managing them.
It's time to experiment with new interpretations of Fit tests. (We still may use Fit to help with this task.) The need is there now, by real people doing real work.
[June, 2005.]