Tag Archives: spreadsheet

Tests from a Hat

How does the order of tests affect the design of software?

Exercise:

  • Write a bunch of plausible unit test descriptions.
  • Slice them into strips and put them in a hat.
  • Repeat until the hat is empty:
    • Pull tests one at a time from a hat.
    • Write the indicated test
    • Write the code to make the last test pass, TDD style.
    • If you absolutely must create a prerequisite test, do so.
    • Refactor as you go.
  • Set aside your implementation, shuffle the tests back into the hat, and repeat once or twice.

Exercise ideas:

  • spreadsheet (domain objects, parsing, representation of storage),
  • auction sniper (from Freeman and Pryce’s Growing Object-Oriented Software),
  • video rental store (from Fowler et al’s Refactoring),
  • currency (from Beck’s Test-Driven Development),

Sample Tests for a Spreadsheet:

  1. New sheet is empty
  2. Inserting into a sheet stores a value
  3. Numeric cells are formatted as numbers
  4. Cells retain literal values (for editing) Ex: 007 is edited as 007 but displays as 7.
  5. Cells starting with = are formulas.
  6. Constant formula Ex. =7
  7. Parentheses in formulas
  8. Multiplication in formulas
  9. Addition in formulas
  10. Precedence in formulas

Reflection Points:

  • Did you encode your tests the same each time?
  • Did the solutions come out basically the same? Why or why not?
  • Did you see alternative solution paths when you went through again?

Source:
Bill Wake, 2010. Inspired by "Scenes From a Hat" in improv.

Fit Spreadsheet – Output

Starting Fit

To use fit, you create a web page that has tables in it; the tables specify tests. (There are other options but that is easiest.) In this case, I'm using Microsoft Word™ and saving the file in HTML format.

 

The fit FileRunner acts as a filter: given a web page, it copies text outside of tables as is, and runs your program on the table entries. Some table entries represent tests that can pass or fail; fit colors them green or red respectively. The output is another HTML file.

 

Fit will also put a summary in the file if you put in a table like this:

 

fit.Summary

counts 0 right, 0 wrong, 0 ignored, 0 exceptions
input file C:\P4\FirstFit\fit\FirstFit-in.htm
input update Thu May 01 10:51:42 EDT 2003
output file C:\P4\FirstFit\fit\FirstFit-out.htm
run date Thu May 01 10:58:28 EDT 2003
run elapsed time 0:00.05

 

With this tool, you don’t manipulate screen elements directly. Instead, you work with an abstraction of them. To me, it feels like talking to somebody over the phone, trying to tell them how to use an application. (“In cell cee seventeen, put equals a one; then go to a one and type ‘fish’.”)

 

This article shows the input to [output from] fit; the result of running it is here [input is here].

Programming and Configuration Notes                                                                

Fit is a tool for customers and testers, but programmers will use it as well, and will have to write some of the fixtures the team uses. In this paper, I’ve tried to use the framework mostly straight out of the box.

 

The CLASSPATH needs to include fit.jar (both in the DOS window and the IDE). The runner command I’m using is:

java fit.FileRunner FirstFit-in.htm FirstFit-out.htm

 

When I do this on the file I have so far, it creates the output file and writes this to the console:

0 right, 0 wrong, 0 ignored, 0 exceptions

Fixtures

Tables in the input file have the name of a fixture in the first row. A fixture is a class that knows how to process the table. Fit comes with several fixtures built in, and programmers can create others.

 

One simple fixture is the ColumnFixture. In this fixture, the first row is the fixture name, and the second row has the names of data. If a name ends without parentheses, it is regarded as a field to fill in; with parentheses, it’s treated as a method (function) call. The fixture fills in all the data fields, and then calls the methods to verify that they return the expected results.

 

Another standard fixture is the ActionFixture. This one consists of a series of commands. These include:

  • start classname: Creates an object of the specified class
  • enter field value: Sets the field to the value
  • press button-name: Calls the method corresponding to the button
  • check method value: Checks that the method returns the expected value

 

The ActionFixture ignores anything past the first three columns; we’ll use the fourth column for comments.

 

So, we’re finally ready to start our application.

 

fit.ActionFixture

start

Spreadsheet

Create a new spreadsheet.

 

This test doesn’t ask for much, but of course it fails. (There isn’t any code yet!)

            0 right, 0 wrong, 0 ignored, 1 exceptions

 

Programmer Notes

The exception is thrown because the Spreadsheet object doesn’t exist. To create it as simply as possible, make it extend Fixture:

 

import fit.Fixture;

public class Spreadsheet extends Fixture {}

 

This gets us back to

            0 right, 0 wrong, 0 ignored, 0 exceptions

 

I’ve put together stubs for the fixtures used in this article: Spreadsheet.java, SpreadsheetFormula.java, and Address.java; here’s a zip file containing all three.

A Few Stories

We have several things we want our spreadsheet to do:

  • Track the contents of cells
  • Distinguish data from formulas
  • Provide both data and formula views of cells
  • Support “+” for appending strings, “’” for reversing strings, “()” for grouping, and “>” for string containment.

Cells

The spreadsheet has a number of cells, each of which has an address. Cells contain string data or formulas.

 

We’ll assume several screen elements:

  • address – the address we’re working with; something like “B19”
  • cell – the cell contents we enter (to the last “address”)
  • formula – the cell contents as entered (for the last “address”)
  • display – the cell contents as seen when the formulas are applied (for the last “address”)

 

We’ll start with a simple data cell.

fit.ActionFixture

Comments

start

Spreadsheet

 

 

enter

a1

abc

 

check

a1

abc

Text in cell

check

formula

abc

Formula is same. (Looks in last-mentioned cell.)

 

Now let’s add in a formula cell. (Note that this table omits the “start” line; this means it’s working on the same object as before. This lets us not repeat the setup, but it also makes the tests less independent.)

 

fit.ActionFixture

Comments

enter

a1

abc

 

enter

b1

=A1

Simple copying formula

check

formula

=A1

Formula is there

check

a1

abc

Original text in A1

check

b1

abc

Text was copied to B1

 

The essence of a spreadsheet is the automatic updates. Let’s change A1 and see it happen.

 

fit.ActionFixture

Comments

enter

a1

abc

 

enter

b1

=A1

Simple copying formula

check

b1

abc

Copied value

enter

a1

revised

Update A1

check

b1

revised

Automatically updates B1

 

 

We already have quite a few elements in use, though we haven’t specified exactly what is valid. Let’s just note the “specification debt” and move on.

  • What can a cell hold? Empty string, other string, formula starts with “=”
  • What’s a valid address? Letter plus digits; ignore leading 0s; case-insensitive.
  • What’s a valid formula? So far, we’ve just used a simple cell reference, but we want operators too.
  • What happens when a cell has an invalid formula?
  • What happens when a cell refers to a cell containing a formula?
  • What happens when formulas form a loop?

 

We’ll pursue all these, but let’s start with formulas.

Formulas

Formulas can reference formulas.

 

SpreadsheetFormula

a1

b1

c1

d1

a1()

b1()

c1()

d1()

data

=A1

=B1

=C1

data

data

data

data

 

  

Formulas get more interesting when there are operators available. The reverse operator (‘) is probably a good one to start with.

SpreadsheetFormula

a1

b1

b1()

abc

=A1'

cba

abc

=A1''''

abc

 

 

 

The most useful string operator is probably append (+):

SpreadsheetFormula

a1

b1

c1

b1()

c1()

abc

=A1+A1

blank

abcabc

 

blank

abc

def

=A1+B1+B1+A1

def

abcdefdefabc

 

 

We have enough features that we can demonstrate an identity: (XY)’=Y’X’. We don’t have parentheses yet, but we can simulate this by putting the parts in separate cells.

 

SpreadsheetFormula

a1

b1

c1

d1

e1

d1()

e1()

abc

xyz

ignored

=A1+B1

=D1'

abcxyz

zyxcba

abc

xyz

=B1'

=A1'

=C1+D1

cba

zyxcba

 

  

Parentheses can be used to group operators. Let’s re-do the previous test, allowing parentheses:

SpreadsheetFormula

a1

b1

c1

c1()

abc

xyz

=(A1+B1)'

zyxcba

abc

xyz

=B1'+A1'

zyxcba

 

 The operator “>” tells whether one string contains another one. If the first string contains the second, the result is the second. If the first string doesn’t contain the second, the result is an empty string.

 

SpreadsheetFormula

a1

b1

c1

c1()

banana

ana

=A1>B1

 ana

banana

bab

=A1>B1

 

blank

 

We haven’t talked about precedence yet. The ‘ and () operators have the highest precedence, then +, then >. A1+B1+C1 is a legal expression, but A1>B1>C1 is not.

 

SpreadsheetFormula

a1

b1

c1

c1()

abc

xyz

=A1+B1'

abczyx

abc

xyz

=(A1+B1)'

zyxcba

 

 

SpreadsheetFormula

a1

b1

c1

d1

e1

e1()

abcdef

ghijkl

e

hgf

=A1+B1>C1+D1'

efgh

 

Backfill

We have several questions left open:

  • What can a cell hold? Empty string, other string, formula starts with “=”
  • What’s a valid address? Letter plus digits; ignore leading 0s; case-insensitive.
  • What happens when a cell has an invalid formula?
  • What happens when formulas form a loop?

 

The previous tests made a quick pass through the system. I think of them as generative: they help define the essence of the system. But questions like the above require us to fill in the gaps. I think of tests that do things like check “corner cases,” error cases, and how features interact as elaborative; they fill in what we already have. They might find problems, but they may well work already, depending on how the system was built.

What a cell holds

We already have test cases where a cell holds a string, and where a cell holds a formula, but it would be prudent to check that the operators work correctly on empty strings. If e is the empty string and x is a non-empty string, we expect:

            e’ = e

            e+e=e

            e+x=x

            x+e=x

            e>e=e

            e>x=e

            x>e=e

 

As I go to write the test, I realize that we never specified what a cell starts with. The answer, of course, is the empty string. So we’ll rely on that: A1 will be empty.

 

fit.ActionFixture

Comments

start

Spreadsheet

 

 

check

a1

 

Verify that cell starts empty.

 

Then we can verify those rules about working with the empty string:

 

SpreadsheetFormula

a1

b1

c1

c1()

Comment

blank

blank

=A1'

blank

e’=e

blank

blank

=A1+A1

blank

e+e=e

blank

blank

=A1>A1

blank

e>e=e

blank

abc

=A1+B1

abc

e+x=x

blank

abc

=B1+A1

abc

x+e=x

blank

abc

=A1>B1

blank

e>x=e

blank

abc

=B1>A1

blank

x>e=e

 

Valid Addresses

There are two places we use addresses: in the address field and in the cells with formulas. When we get a “real” (graphical) interface, the address will mostly be implicit. But even so, we’ll test it here just to be safe.

 

Let’s use a ColumnFixture for this: we’ll put address in one column, valid() in another, and standardized() in another. (A programmer will have to write the new fixture for us.)

 

The rules are: a valid address is a letter (A-Z, a-z) followed by one or more digits (0-9). Case is ignored. Leading 0s are ignored. “0” is not a valid row number.

 

Address

address

valid()

standardized()

A1

true

A1

a1

true

A1

A9874

true

A9874

Z1

true

Z1

z1

true

Z1

Z3992

true

Z3992

z3992

true

Z3992

AA393

false

 

zX202

false

 

é17

false

 

1

false

 

~1

false

 

~D1

false

 

y&1

false

 

^

false

 

X392%

false

 

H001

true

H1

j00010

true

J10

e000

false

 

A0

false

 

z0

false

 

 

Let’s make sure that case-insensitivity works in formulas:

 

SpreadsheetFormula

a1

b1

b1()

abc

=A1+a1

abcabc

 

Formula Errors

If a formula contains an error, we’d like it to display as “#error.” We’ll put all the invalid names from the previous table into formulas, and verify that formulas behave correctly. Then we’ll try various improper combinations of operators.

fit.ActionFixture

start

Spreadsheet

Create a new spreadsheet.

enter

a1

=AA393

Bad address

check

a1

#error

Marked as error

check

formula

=AA393

Formula as written

enter

a1

=A2

Change to valid address

check

a1

 

Make sure #error is cleared 

 

 

SpreadsheetFormula

a1

a1()

Comment

=zX202

#error

Two letters

=é17

#error

expected


? actual

Non-ASCII

=1

#error

No letters

=~1

#error

No letters

=~D1

#error

Unacceptable character

=y&1

#error

Extra character

=^

#error

No letters/digits

=e000

#error

expected


? actual

Too many digits

=A0

#error

expected


? actual

Invalid row #

=z0

#error

expected


? actual

Invalid row #

=

#error

Missing formula

  

Then we’ll get to some operators:

SpreadsheetFormula

a1

a1()

Comment

='A2

#error

'  should be postfix

='A2'

#error

Can’t be before and after

=A2+

#error

Need other term

=A3+A4+

#error

Need other term

=A2++A3

#error

Missing term

=A2+'+A3

#error

‘ isn’t a term

=A2'''+A3

blank

OK to mix things

=A2)

#error

Missing (

=(A2

#error

Missing (

=((((((((((((A2))))))))))))

blank

OK – big expression

=((((((A2+(A3))))+A4)

#error

Unbalanced – too few )

=(((A2>A3

#error

Unbalanced – too few )

=(A2>A3)))

#error

Unbalanced – too many )

=A2>A3>

#error

Can’t trail >

=A2>A3>A4

#error

Can’t repeat >

 

Loops

If a formula uses itself (directly or indirectly), we don’t want it to loop forever trying to figure it out. Instead, we’d like the display to be “#loop.”

 

SpreadsheetFormula

a1

b1

c1

d1

e1

a1()

e1()

=A1

blank

blank

blank

blank

#loop

blank

=B1

=C1

=F1+D1

=E1

no-loop

no-loop

no-loop

=B1

=C1

=F1+D1

=E1

=A1

#loop

#loop

 

Conclusions

This paper has demonstrated a set of tests using the fit acceptance testing framework. Some things to note:

  • The tests here have been written as if a customer specified them, without much demonstration of the programming cycle. But programmers can work with these tests in much the way they would with JUnit.
  • The tests are written without benefit of the feedback of a working system. (I wrote just enough code to make the first test not throw an exception.)
  • The tests look at only part of the system: the core functionality. There are other aspects of a real application that we aren’t testing. (For example, it may be non-trivial to connect a screen to the core code.)
  • Even a small application such as this requires a fairly large set of tests. With more programming work on the fixtures, we might be able to reduce some of the noise. Real applications will organize tests into multiple files, and will have to pay more attention to the challenges of consistency, test independence, and feature interaction.
  • It feels smooth to mix light natural-language specification with formal, executable tests.
  • Fit has a number of features we haven’t used.

 

I’ve heard that many teams use xUnit for unit testing, but still struggle to get customer tests before or even after stories are implemented. I hope frameworks such as fit can help lower the barriers to doing this crucial task. 

 

fit.Summary

counts 94 right, 4 wrong, 0 ignored, 0 exceptions
input file C:\P4\FirstFit\fit\FirstFit-in.htm
input update Thu May 01 10:51:42 EDT 2003
output file C:\P4\FirstFit\fit\FirstFit-out.htm
run date Thu May 01 10:58:28 EDT 2003
run elapsed time 0:00.14

 

Resources and Related Articles

 

[Written April 20, 2003; revised April 26, 2003, to correct mis-stated identity & in response to Ward Cunningham's great suggestions about improving the fixtures; 2012 – the WordPress version is designed to simulate the original look.]

Fit Spreadsheet

Ward Cunningham has created an acceptance testing framework known as fit. (See http://fit.c2.com for more details.) In this brief experiment, we'll use tests to help specify a simple spreadsheet for strings.

Starting Fit

To use fit, you create a web page that has tables in it; the tables specify tests. (There are other options but that is easiest.) In this case, I'm using Microsoft Word(tm) and saving the file in HTML format.

The fit FileRunner acts as a filter: given a web page, it copies text outside of tables as is, and runs your program on the table entries. Some table entries represent tests that can pass or fail; fit colors them green or red respectively. The output is another HTML file.

Fit will also put a summary in the file if you put in a table like this:

fit.Summary

With this tool, you don't manipulate screen elements directly. Instead, you work with an abstraction of them. To me, it feels like talking to somebody over the phone, trying to tell them how to use an application. ("In cell cee seventeen, put equals a one; then go to a one and type 'fish'.")

This article shows the input to fit; the result of running it is here.

Programming and Configuration Notes

Fit is a tool for customers and testers, but programmers will use it as well, and will have to write some of the fixtures the team uses. In this paper, I've tried to use the framework mostly straight out of the box.

The CLASSPATH needs to include fit.jar (both in the DOS window and the IDE). The runner command I'm using is:

java fit.FileRunner FirstFit-in.htm FirstFit-out.htm

When I do this on the file I have so far, it creates the output file and writes this to the console:

0 right, 0 wrong, 0 ignored, 0 exceptions

Fixtures

Tables in the input file have the name of a fixture in the first row. A fixture is a class that knows how to process the table. Fit comes with several fixtures built in, and programmers can create others.

One simple fixture is the ColumnFixture. In this fixture, the first row is the fixture name, and the second row has the names of data. If a name ends without parentheses, it is regarded as a field to fill in; with parentheses, it's treated as a method (function) call. The fixture fills in all the data fields, and then calls the methods to verify that they return the expected results.

Another standard fixture is the ActionFixture. This one consists of a series of commands. These include:

  • start classname: Creates an object of the specified class
  • enter field value: Sets the field to the value
  • press button-name: Calls the method corresponding to the button
  • check method value: Checks that the method returns the expected value

The ActionFixture ignores anything past the first three columns; we'll use the fourth column for comments.

So, we're finally ready to start our application.

fit.ActionFixture
start Spreadsheet Create a new spreadsheet.

This test doesn't ask for much, but of course it fails. (There isn't any code yet!)

            0 right, 0 wrong, 0 ignored, 1 exceptions

Programmer Notes

The exception is thrown because the Spreadsheet object doesn't exist. To create it as simply as possible, make it extend Fixture:

import fit.Fixture;

public class Spreadsheet extends Fixture {}

This gets us back to

            0 right, 0 wrong, 0 ignored, 0 exceptions

I've put together stubs for the fixtures used in this article: Spreadsheet.java, SpreadsheetFormula.java, and Address.java; here's a zip file containing all three.

A Few Stories

We have several things we want our spreadsheet to do:

  • Track the contents of cells
  • Distinguish data from formulas
  • Provide both data and formula views of cells
  • Support "+" for appending strings, "'" for reversing strings, "()" for grouping, and ">" for string containment.

Cells

The spreadsheet has a number of cells, each of which has an address. Cells contain string data or formulas.

We'll assume several screen elements:

  • a1 – the cell "A1". For "enter," we'll put something in the cell; for "check," we'll get its displayed value.
  • b1 – the same for cell "B1".
  • formula – the formula of the last-mentioned cell.

We'll start with a simple data cell.

fit.ActionFixture Comments
start Spreadsheet    
enter a1 abc  
check a1 abc Text in cell
check formula abc Formula is same. (Looks in last-mentioned cell.)

Now let's add in a formula cell. (Note that this table omits the "start" line; this means it's working on the same object as before. This lets us not repeat the setup, but it also makes the tests less independent.)

fit.ActionFixture Comments
enter a1 abc  
enter b1 =A1 Simple copying formula
check formula =A1 Formula is there
check a1 abc Original text in A1
check b1 abc Text was copied to B1

The essence of a spreadsheet is the automatic updates. Let's change A1 and see it happen.

fit.ActionFixture Comments
enter a1 abc  
enter b1 =A1 Simple copying formula
check b1 abc Copied value
enter a1 revised Update A1
check b1 revised Automatically updates B1

We already have quite a few elements in use, though we haven't specified exactly what is valid. Let's just note the "specification debt" and move on.

  • What can a cell hold? Empty string, other string, formula starts with "="
  • What's a valid address? Letter plus digits; ignore leading 0s; case-insensitive.
  • What's a valid formula? So far, we've just used a simple cell reference, but we want operators too.
  • What happens when a cell has an invalid formula?
  • What happens when a cell refers to a cell containing a formula?
  • What happens when formulas form a loop?

We'll pursue all these, but let's start with formulas.

Formulas

Formulas can reference formulas. We'll use a new ColumnFixture, SpreadsheetFormula, that lets us specify the inputs and expected outputs of cells. This fixture should access the same type spreadsheet as used by Spreadsheet.

SpreadsheetFormula
a1 b1 c1 d1 a1() b1() c1() d1()
data =A1 =B1 =C1 data data data data

Formulas get more interesting when there are operators available. The reverse operator (') is probably a good one to start with.

SpreadsheetFormula
a1 b1 b1()
abc =A1' cba
abc =A1'''' abc

The most useful string operator is probably append (+). Fit ignores input cells that are left blank, so we'll explicitly use the word "blank" when we want an empty cell. The fixture will have to take this into account.

SpreadsheetFormula
a1 b1 c1 b1() c1()
abc =A1+A1 blank abcabc  
abc def =A1+B1+B1+A1 def abcdefdefabc

We have enough features that we can demonstrate an identity: (XY)'=Y'X'. We don't have parentheses yet, but we can simulate this by putting the parts in separate cells.

SpreadsheetFormula
a1 b1 c1 d1 e1 d1() e1()
abc xyz ignored =A1+B1 =D1' abcxyz zyxcba
abc xyz =B1' =A1' =C1+D1 cba zyxcba

Parentheses can be used to group operators. Let's re-do the previous test, allowing parentheses:

SpreadsheetFormula
a1 b1 c1 c1()
abc xyz =(A1+B1)' zyxcba
abc xyz =B1'+A1' zyxcba

The operator ">" tells whether one string contains another one. If the first string contains the second, the result is the second. If the first string doesn't contain the second, the result is an empty string.

SpreadsheetFormula
a1 b1 c1 c1()
banana ana =A1>B1  ana
banana bab =A1>B1  

We haven't talked about precedence yet. The ' and () operators have the highest precedence, then +, then >. A1+B1+C1 is a legal expression, but A1>B1>C1 is not.

SpreadsheetFormula
a1 b1 c1 c1()
abc xyz =A1+B1' abczyx
abc xyz =(A1+B1)' zyxcba

 

SpreadsheetFormula
a1 b1 c1 d1 e1 e1()
abcdef ghijkl e hgf =A1+B1>C1+D1' efgh

Filling in the Gaps

We have several questions left open:

  • What can a cell hold? Empty string, other string, formula starts with "="
  • What's a valid address? Letter plus digits; ignore leading 0s; case-insensitive.
  • What happens when a cell has an invalid formula?
  • What happens when formulas form a loop?

The previous tests made a quick pass through the system. I think of them as generative: they help define the essence of the system. But questions like the above require us to fill in the gaps. I think of tests that do things like check "corner cases," error cases, and how features interact as elaborative; they fill in what we already have. They might find problems, but they may well work already, depending on how the system was built.

What a cell holds

We already have test cases where a cell holds a string, and where a cell holds a formula, but it would be prudent to check that the operators work correctly on empty strings. If e is the empty string and x is a non-empty string, we expect:

            e' = e
            e+e=e
            e+x=x
            x+e=x
            e>e=e
            e>x=e
            x>e=e

As I go to write the test, I realize that we never specified what a cell starts with. The answer, of course, is the empty string. So we'll rely on that: A1 will be empty.

fit.ActionFixture Comments
start Spreadsheet    
check a1   Verify that cell starts empty.

  Then we can verify those rules about working with the empty string:

SpreadsheetFormula
a1 b1 c1 c1() Comment
blank blank =A1' blank e'=e
blank blank =A1+A1 blank e+e=e
blank blank =A1>A1 blank e>e=e
blank abc =A1+B1 abc e+x=x
blank abc =B1+A1 abc x+e=x
blank abc =A1>B1 blank e>x=e
blank abc =B1>A1 blank x>e=e

  Valid Addresses

There are two places we use addresses: in the address field and in the cells with formulas. When we get a "real" (graphical) interface, the address will mostly be implicit. But even so, we'll test it here just to be safe.

Let's introduce a new fixture, Address. It will be a ColumnFixture: we'll put address in one column, valid() in another, and standardized() in another. (A programmer will have to write the new fixture for us.)

The rules are: a valid address is a letter (A-Z, a-z) followed by one or more digits (0-9). Case is ignored. Leading 0s are ignored. "0" is not a valid row number.

Address
address valid() standardized()
A1 true A1
a1 true A1
A9874 true A9874
Z1 true Z1
z1 true Z1
Z3992 true Z3992
z3992 true Z3992
AA393 false  
zX202 false  
é17 false  
1 false  
~1 false  
~D1 false  
y&1 false  
^ false  
X392% false  
H001 true H1
j00010 true J10
e000 false  
A0 false  
z0 false  

Let's make sure that case-insensitivity works in formulas:

SpreadsheetFormula
a1 b1 b1()
abc =A1+a1 abcabc

Formula Errors

If a formula contains an error, we'd like it to display as "#error." We'll put all the invalid names from the previous table into formulas, and verify that formulas behave correctly. Then we'll try various improper combinations of operators.

fit.ActionFixture
start Spreadsheet Create a new spreadsheet.
enter a1 =AA393 Bad address
check a1 #error Marked as error
check formula =AA393 Formula as written
enter a1 =A2 Change to valid address
check a1   Make sure #error is cleared 

   

SpreadsheetFormula
a1 a1() Comment
=zX202 #error Two letters
=é17 #error Non-ASCII
=1 #error No letters
=~1 #error No letters
=~D1 #error Unacceptable character
=y&1 #error Extra character
=^ #error No letters/digits
=e000 #error Too many digits
=A0 #error Invalid row #
=z0 #error Invalid row #
= #error Missing formula

 Then we'll get to some operators:

SpreadsheetFormula
a1 a1()

Comment

='A2 #error '  should be postfix
='A2' #error Can't be before and after
=A2+ #error Need other term
=A3+A4+ #error Need other term
=A2++A3 #error Missing term
=A2+'+A3 #error ' isn't a term
=A2'''+A3 blank OK to mix things
=A2) #error Missing (
=(A2 #error Missing (
=((((((((((((A2)))))))))))) blank OK – big expression
=((((((A2+(A3))))+A4) #error Unbalanced – too few )
=(((A2>A3 #error Unbalanced – too few )
=(A2>A3))) #error Unbalanced – too many )
=A2>A3> #error Can't trail >
=A2>A3>A4 #error Can't repeat >

Loops

If a formula uses itself (directly or indirectly), we don't want it to loop forever trying to figure it out. Instead, we'd like the display to be "#loop."

SpreadsheetFormula
a1 b1 c1 d1 e1 a1() e1()
=A1 blank blank blank blank #loop blank
=B1 =C1 =F1+D1 =E1 no-loop no-loop no-loop
=B1 =C1 =F1+D1 =E1 =A1 #loop #loop

Conclusions

This paper has demonstrated a set of tests using the fit acceptance testing framework. Some things to note:

  • The tests here have been written as if a customer specified them, without much demonstration of the programming cycle. But programmers can work with these tests in much the way they would with JUnit.
  • The tests are written without benefit of the feedback of a working system. (I wrote just enough code to make the first test not throw an exception.) When I went back to implement the system, I found a number of bugs in the tests.
  • The tests look at only part of the system: the core functionality. There are other aspects of a real application that we aren't testing. (For example, it may be non-trivial to connect a screen to the core code.)
  • Even a small application such as this requires a fairly large set of tests. With more programming work on the fixtures, we might be able to reduce some of the noise. Real applications will organize tests into multiple files, and will have to pay more attention to the challenges of consistency, test independence, and feature interaction.
  • It feels smooth to mix light natural-language specification with formal, executable tests.
  • Fit has a number of features we haven't used.

I've heard that many teams use xUnit for unit testing, but still struggle to get customer tests before or even after stories are implemented. I hope frameworks such as fit can help lower the barriers to doing this crucial task. 

fit.Summary

 

Resources and Related Articles

[Written April 20, 2003; revised April 26, 2003, to correct mis-stated identity & in response to Ward Cunningham's great suggestions about improving the fixtures. Revised May 1, 2003 to fix some test problems. 2012 – the WordPress version is designed to simulate the original look.]

Test-First Challenge

This exercise offers you a chance to practice test-first through a series of challenges.

Each challenge provides a set of tests. Your job is to write code that supports the tests.

My goals are:

  1. To give people learning or practicing test-first a way to begin small-step incremental programming, without the pressure of having to invent tests.
  2. To find out how similar the designs will be.

The original form of the challenges were posted on two egroups: extremeprogramming (for regular readers there) and xplorations (for low volume). The tests are in Java, but should have an obvious translation to other languages.

Problem Domain

For the problem, I'll use a (super-) simple spreadsheet. My plan is to explore formulas and dependencies before dealing with any user interface.

What To Do

  • Implement code to meet the tests, one at a time. (No fair peeking ahead:)
  • You may feel the need to add more tests than I've given, either to explore an area more thoroughly, or to let yourself take smaller steps. This is a good impulse – add the tests you want.
  • Refactor after each test.
  • Reflect on the process
    • Are these the tests you'd have written?
    • Did you create the simplest code you could?
    • Did you spend much time debugging?
    • Any surprises in how your system is structured?
  • (Optional) After you've done a particular challenge, I'd love to hear from you. Email to William.Wake@acm.org. If you want to make life easy and interesting for me, make the message like this:
    Subject: CHALLENGE PART n
    Body: Brief description of the decisions you made ("I used a 
    Frobble to calculate interest, and I split Foos into Bars and Bazs.")
    Attachment: A Zip file of your source code, named TFn.zip.

    I'd be glad to hear your design decisions and comments even if you don't want to send your code. I won't re-post anything you send me without your permission.


Challenge, Part 1

Domain: A simple spreadsheet with columns A..Z, AA.., and rows 1..n.

Challenge Tests

public void testThatCellsAreEmptyByDefault() {
  Sheet sheet = new Sheet();
  assertEquals("", sheet.get("A1"));
  assertEquals("", sheet.get("ZX347"));
}

// Implement each test before going to the next one.

public void testThatTextCellsAreStored() {
  Sheet sheet = new Sheet();
  String theCell = "A21";

  sheet.put(theCell, "A string");
  assertEquals("A string", sheet.get(theCell));

  sheet.put(theCell, "A different string");
  assertEquals("A different string", sheet.get(theCell));

  sheet.put(theCell, "");
  assertEquals("", sheet.get(theCell));
}

// Implement each test before going to the next one; then refactor.

public void testThatManyCellsExist() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "First");
  sheet.put("X27", "Second");
  sheet.put("ZX901", "Third");

  assertEquals("A1", "First", sheet.get("A1"));
  assertEquals("X27", "Second", sheet.get("X27"));
  assertEquals("ZX901", "Third", sheet.get("ZX901"));

  sheet.put("A1", "Fourth");
  assertEquals("A1 after", "Fourth", sheet.get("A1"));
  assertEquals("X27 same", "Second", sheet.get("X27"));
  assertEquals("ZX901 same", "Third", sheet.get("ZX901"));
}


// Implement each test before going to the next one.
// You can split this test case if it helps.

public void testThatNumericCellsAreIdentifiedAndStored() {
  Sheet sheet = new Sheet();
  String theCell = "A21";

  sheet.put(theCell, "X99"); // "Obvious" string
  assertEquals("X99", sheet.get(theCell));

  sheet.put(theCell, "14"); // "Obvious" number
  assertEquals("14", sheet.get(theCell));

  sheet.put(theCell, " 99 X"); // Whole string must be numeric
  assertEquals(" 99 X", sheet.get(theCell));

  sheet.put(theCell, " 1234 "); // Blanks ignored
  assertEquals("1234", sheet.get(theCell));

  sheet.put(theCell, " "); // Just a blank
  assertEquals(" ", sheet.get(theCell));
}

// Refactor before going to each succeeding test.

public void testThatWeHaveAccessToCellLiteralValuesForEditing() { 
  Sheet sheet = new Sheet();
  String theCell = "A21";

  sheet.put(theCell, "Some string"); 
  assertEquals("Some string", sheet.getLiteral(theCell));

  sheet.put(theCell, " 1234 "); 
  assertEquals(" 1234 ", sheet.getLiteral(theCell));

  sheet.put(theCell, "=7"); // Foreshadowing formulas:)
  assertEquals("=7", sheet.getLiteral(theCell));
}


// We'll talk about "get" and formulas next time.

Challenge, Part 2

What To Do

  • Do part 1 if you haven't yet.
  • Implement code for each test (one at a time), then refactor to simplify your code.
  • (Optional) Email me with results and feedback. (William.Wake@acm.org)

Challenge Tests

// Implement code for previous test before moving to next one.

public void testFormulaSpec() {
  Sheet sheet = new Sheet();
  sheet.put("B1", " =7"); // note leading space
  assertEquals("Not a formula", " =7", sheet.get("B1"));
  assertEquals("Unchanged", " =7", sheet.getLiteral("B1"));
}

// Next - start on parsing expressions

public void testConstantFormula() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=7");
  assertEquals("Formula", "=7", sheet.getLiteral("A1"));
  assertEquals("Value", "7", sheet.get("A1"));
}

// More formula tests. You may feel the need to make up 
// additional intermediate test cases to drive your code
// better. (For example, you might want to test "2*3" 

// before "2*3*4".) That's fine, go ahead and create them.
// Just keep moving one test at a time.

// We're doing expressions; you may need to do a spike
// (investigation) if you're not familiar with parsing.
// For background, look up "recursive descent" or
// "operator precedence". (Other techniques can work as well.)

// Order of tests - I'm familiar enough with parsing to think
// it's probably easiest to do them in this order (highest
// precedence to lowest). For extra credit, you might redo 
// this part of the exercise with the tests in a different order 
// to see what difference it makes.

public void testParentheses() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=(7)");
  assertEquals("Parends", "7", sheet.get("A1"));
}

public void testDeepParentheses() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=((((10))))");
  assertEquals("Parends", "10", sheet.get("A1"));
}

public void testMultiply() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=2*3*4");
  assertEquals("Times", "24", sheet.get("A1"));
}

public void testAdd() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=71+2+3");
  assertEquals("Add", "76", sheet.get("A1"));
}

public void testPrecedence() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=7+2*3");
  assertEquals("Precedence", "13", sheet.get("A1"));
}

public void testFullExpression() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=7*(2+3)*((((2+1))))");
  assertEquals("Expr", "105", sheet.get("A1"));
}

// Add any test cases you feel are missing based on 
// where your code is now.

// Then try your hand at a few test cases: Add "-" and "/"

// with normal precedence. 

// Next, error handling.

public void testSimpleFormulaError() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=7*");
  assertEquals("Error", "#Error", sheet.get("A1"));
}

public void testParenthesisError() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=(((((7))");
  assertEquals("Error", "#Error", sheet.get("A1")); 
}

// Add any more error cases you need. Numeric errors (e.g.,
// divide by 0) can return #Error too.

// Take a deep breath and refactor. This was a big jump.
// Next time we'll tackle formulas involving cells.

Challenge, Part 3

We're going to add dependencies now. This is one of those things that makes a spreadsheet a spreadsheet.

What To Do

  • Do parts 1 and 2 if you haven't yet.
  • Implement code for each test (one at a time), then refactor to simplify your code.
  • (Optional) Email me with results and feedback. (William.Wake@acm.org)

Challenge Tests

public void testThatCellReferenceWorks () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "8");
  sheet.put("A2", "=A1");
  assertEquals("cell lookup", "8", sheet.get("A2"));
}

public void testThatCellChangesPropagate () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "8");
  sheet.put("A2", "=A1");
  assertEquals("cell lookup", "8", sheet.get("A2"));

  sheet.put("A1", "9");
  assertEquals("cell change propagation", "9", sheet.get("A2"));
}

public void testThatFormulasKnowCellsAndRecalculate () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "8");
  sheet.put("A2", "3");
  sheet.put("B1", "=A1*(A1-A2)+A2/3");
  assertEquals("calculation with cells", "41", sheet.get("B1"));

  sheet.put("A2", "6");
  assertEquals("re-calculation", "18", sheet.get("B1"));
}

public void testThatDeepPropagationWorks () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "8");
  sheet.put("A2", "=A1");
  sheet.put("A3", "=A2");
  sheet.put("A4", "=A3");
  assertEquals("deep propagation", "8", sheet.get("A4"));

  sheet.put("A2", "6");
  assertEquals("deep re-calculation", "6", sheet.get("A4"));
}


// The following test is likely to pass already.
public void testThatFormulaWorksWithManyCells () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "10");
  sheet.put("A2", "=A1+B1");
  sheet.put("A3", "=A2+B2");
  sheet.put("A4", "=A3");
  sheet.put("B1", "7");
  sheet.put("B2", "=A2");
  sheet.put("B3", "=A3-A2");
  sheet.put("B4", "=A4+B3");

  assertEquals("multiple expressions - A4", "34", sheet.get("A4"));
  assertEquals("multiple expressions - B4", "51", sheet.get("B4"));
}

// Refactor and get everything nice and clean.


// Next: (I almost made this a separate part, and when I
// originally did it, I did it in a different design session).
// So take a break if you need one.

// There's one big open issue for formulas: what about
// circular references? 

// I'll sketch some hints, but you should define your own tests
// that drive toward a solution compatible with your own 
// implementation.

public void testThatCircularReferenceDoesntCrash() {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=A1");
  assertTrue(true);
}

// Just like errors return a special value, it might be nice
// if circular references did too. (See notes below).

public void testThatCircularReferencesAdmitIt () {
  Sheet sheet = new Sheet();
  sheet.put("A1", "=A1");
  assertEquals("Detect circularity", "#Circular", sheet.get("A1"));
}

// You might come up with some other approach that suits your 
// taste. We won't be exploring this corner of the solution 
// any further; you just want a scheme that blocks silly mistakes.
// Make sure you test deep circularities involving partially
// evaluated expressions.

// A hint: if you blindly evaluate an expression you have no
// control over how deep the expression can be, since
// circular references appear to be infinitely deep.


// Where are we? I intend to spend the next two parts hooking
// up a GUI. Then there will be an optional part that pushes 
// things in an unexpected direction just to get a sense
// of our software's robustness.

Part 4, Moving Toward a GUI

What To Do

  • Do parts 1 through 3 if you haven't yet.
  • Implement code for each test (one at a time), then refactor to simplify your code.
  • (Optional) Email me with results and feedback. (William.Wake@acm.org)

Challenge Tests

We'll start by using a feature of JUnit that we haven't used before: a setUp() method. To use this, declare your variables at the class level, and initialize them in the setUp() method. There's a corresponding tearDown() method that we won't need, but which can be useful for some tests. JUnit works like this: call setUp(), call one of your tests, call tearDown(), call setUp(), call your next test, call tearDown(), etc. So, each test can assume it's got things just as setUp() left them.

Quick Design

The easiest thing to use in the GUI will be a JTable, which provides the 2-d grid we expect to see. The best way to use a JTable is usually to give it a TableModel. The TableModel acts as a wrapper between the GUI-oriented JTable and your object.

The easiest way to create a TableModel is to subclass AbstractTableModel, so that's what we'll do. The TableModel is 0-based, so we'll use column 0 to hold the row's index, and we'll store spreadsheet row n in TableModel's row n-1.

Sheet sheet;
TableModel table;

public void setUp() {
  sheet = new Sheet();
  table = new SheetTableModel (sheet);        
}

// As usual, do one test at a time and refactor after each.

// For now, we're willing to hard-code a maximum spreadsheet size.
// A future story can deal with this.

int LAST_COLUMN_INDEX = 49;
int LAST_ROW_INDEX = 99;
    
public void testTableModelRequiredOverrides() {
  assertTrue (table.getColumnCount() > LAST_COLUMN_INDEX);
  assertTrue (table.getRowCount() > LAST_ROW_INDEX);
  assertEquals ("", table.getValueAt(10,10));
}

// Take a look at AbstractTableModel's documentation before doing this test.

public void testColumnNames() {
  assertEquals ("", table.getColumnName(0));
  assertEquals ("A", table.getColumnName(1));
  assertEquals ("Z", table.getColumnName(26));
  assertEquals ("AW", table.getColumnName(LAST_COLUMN_INDEX));
}

public void testThatColumn0ContainsIndex() {
  assertEquals ("1", table.getValueAt(0,0));
  assertEquals ("50", table.getValueAt(49, 0));
  assertEquals ("100", table.getValueAt(LAST_ROW_INDEX,0));
}

// Remember, one test at a time, followed by refactoring.

public void testThatMainColumnsHaveContents() {
  sheet.put ("A1", "upper left");
  assertEquals ("upper left", table.getValueAt(0,1));

  sheet.put ("A100", "lower left");
  assertEquals ("lower left", table.getValueAt(LAST_ROW_INDEX, 1));

  sheet.put ("AW1", "upper right");
  assertEquals ("upper right", table.getValueAt(0, LAST_COLUMN_INDEX));

  sheet.put ("AW100", "lower right");
  assertEquals ("lower right", table.getValueAt(LAST_ROW_INDEX, LAST_COLUMN_INDEX));
}

public void testThatStoresWorkThroughTableModel() {
  table.setValueAt("21", 0, 1);
  table.setValueAt("=A1", 1, 1);

  assertEquals("21", table.getValueAt(0,1));
  assertEquals("21", table.getValueAt(1,1));

  table.setValueAt("22", 0, 1);
  assertEquals("22", table.getValueAt(0,1));
  assertEquals("22", table.getValueAt(1,1));
}


// We've established that the table model can get and set values.
// But JTable uses an event notification mechanism to find out
// about the changes.

// To test this, we'll introduce a test helper class. It's a very
// simple listener, and will assure us that notifications are
// sent when changes are made.

// There's a couple of design decisions implicit here. One is that
// we won't attempt to be specific about which cells change; we'll
// just say that the table data has changed and let JTable refresh
// its view of whichever cells it wants. (Because of cell dependencies,
// changes in one cell could potentially no others, all others,
// or anything in between.) We might revisit this decision during
// performance tuning, and try to issue finer-grained notifications.

// The other decision is that we have no mechanism for our Sheet
// to tell the table model about changes. So changes will either need
// to come in through the table model, or we'll have to add some
// notification mechanism to Sheet. For now, just make changes through the table model.

   public class TestTableModelListener implements TableModelListener {
      public boolean wasNotified = false;
        
      public void tableChanged(TableModelEvent e) {wasNotified = true;}
   }
    
public void testThatTableModelNotifies() {
  TestTableModelListener listener = new TestTableModelListener();
  table.addTableModelListener (listener);
  assertTrue (!listener.wasNotified);

  table.setValueAt("22", 0, 1);

  assertTrue (listener.wasNotified);
}


// Note the cast in our test here. Previous tests have been straight
// implementations of TableModel functions; now we're saying that 
// our model has some extra functions. We'll face a small tradeoff later
// when we want access to the feature: if we get the model back from JTable,
// we'll have to cast it; if we don't want to cast it we'll have to
// track it somewhere.

public void testThatSheetTableModelCanGetLiteral() {
  sheet.put("A1", "=7");
  String contents = ((SheetTableModel)table).getLiteralValueAt(0, 1);

  assertEquals("=7", contents);
}

// We've left isCellEditable() false, on the assumption that the way to edit
// the cell is to go to a textbox provided for that purpose (rather than
// in place).

Part 5, GUI

One of the first XPlorations articles I wrote was to address the question, "Can you do a GUI test-first?" My answer there was, "Yes, but…" and this exercise might lead you to the same opinion.

The goal is to make a simple GUI that looks sort of like this:

+--------+---------------+------+
  | Label  |  Text entry   |  OK  |
  +--------+---------------+------+
  |                              ^|
  |        Grid (JTable)         ||
  |                              v|
  +-------------------------------+

The idea is that if you click in the grid, the label tells which cell you're in, and the text entry field contains the literal value of that cell. You can edit it, and when you click "OK", it puts the value back in the cell. Then the whole spreadsheet updates accordingly. With this, you should have a minimalist but working spreadsheet.

Quick Design

I'll put everything in a subclass of JFrame, using the Swing objects JLabel, JTextField, JButton, and JTable. In my frame, I'll have global variables for each of these components. (I won't bother exposing intermediate panels or other stuff not important outside the class.)

Most of the problem will be setup and hookup. I'll want to hook the table to the model, and set up listeners so that the label and textfield know when the selection has changed. I'll need a listener on the ok button to tell when to update the grid with the edited value. And all these objects have various configuration options to set up as well.

Timothy Wall has pointed out that there are tools designed to test GUIs: http://abbot.sf.net, http://jemmy.netbeans.org, and http://jfcunit.sf.net. You might want to give one of these a try and compare the result.

Challenge Tests

Here are my declarations and setUp routine for my test now:

Sheet sheet;
TableModel table;
SheetTableModel model;  // New for part 5
SheetFrame frame;       // New for part 5

public void setUp() {
  sheet = new Sheet();
  table = new SheetTableModel (sheet);  
  
  model = new SheetTableModel(sheet);
  frame = new SheetFrame(model);
}

// Here are the tests:

public void testThatFrameHasRightParts () {
  assertNotNull(frame.table);
  assertNotNull(frame.label);
  assertNotNull(frame.editor);
  assertNotNull(frame.okButton);
  assertSame(model, frame.table.getModel());
}

public void testThatRowAndColumnSelectionAllowed() {
   assertTrue(frame.table.getRowSelectionAllowed());
   assertTrue(frame.table.getColumnSelectionAllowed());
}

    public class TestSelectionListener implements ListSelectionListener {
        public boolean wasNotified = false;
        
        public TestSelectionListener() {}
    
        public void valueChanged(ListSelectionEvent e) {
            wasNotified = true;            
        }      
    }
    
// I expect this test to pass; it verifies how I think listeners work.
// You might call it a spike and omit it.

public void testThatSelectionsNotifyListeners() {
    TestSelectionListener listener = new TestSelectionListener();
    frame.table.getSelectionModel().addListSelectionListener(listener);

    assertTrue (!listener.wasNotified);  

    frame.table.changeSelection (3, 2, false, false);
    
    assertTrue (listener.wasNotified);


    listener.wasNotified = false;
    frame.table.changeSelection (1, 1, false, false);
    assertTrue (listener.wasNotified);
}

// If you need info on hooking up a selection listener, see 
// http://java.sun.com/docs/books/tutorial/uiswing/components/table.html#selection

public void testThatLabelIsUpdatedWhenSelectionChanges() {
    assertEquals("", frame.label.getText());
    
    frame.table.changeSelection (0, 1, false, false);
    assertEquals("A1", frame.label.getText());

    frame.table.changeSelection (10, 10, false, false);
    assertEquals("J11", frame.label.getText());
}

// You might add a main() routine to SheetFrame and
// see how the GUI is looking.

public void testThatEditorSeesLiteralValue() {
    model.setValueAt("=7", 1, 1);
    frame.table.changeSelection(1,1,false,false);
    
    assertEquals("=7", frame.editor.getText());
}


// We would like to have a way to programmatically let the 
// text field click "Enter", but I don't see a mechanism.
// So we'll use the okButton instead.

public void testThatEditedValueGetsSaved() {
    model.setValueAt("=7", 1, 1);
    frame.table.changeSelection(1,1,false,false);

    frame.editor.setText("=8");
    frame.okButton.doClick();
    assertEquals("=8", frame.model.getLiteralValueAt(1,1));
    assertEquals("8", frame.model.getValueAt(1,1));
}


public void testThatValuePropagationWorks () {
  frame.model.setValueAt("7", 0,1);
  frame.model.setValueAt("=A1+2", 2,2);
  assertEquals("9", frame.model.getValueAt(2,2));
  assertEquals("=A1+2", frame.model.getLiteralValueAt(2,2));
  
  frame.model.setValueAt("10", 0,1);
  assertEquals("12", frame.model.getValueAt(2,2));
}


// See discussion below on acceptance tests.
public void testAcceptanceTest1() {
    SheetTableModel model;
    SheetFrame frame;

    model = new SheetTableModel(new Sheet());
    frame = new SheetFrame(model);
    
    frame.table.changeSelection(0,1,false,false);   // A1
    frame.editor.setText("8");
    frame.okButton.doClick();

    frame.table.changeSelection(1,1,false,false);   // A2
    frame.editor.setText("=A1*A1+A1");
    frame.okButton.doClick();
    
    assertEquals("72", frame.table.getValueAt(1,1));
    
    frame.table.changeSelection(0,1,false,false);   // A1
    frame.editor.setText("5");
    frame.okButton.doClick();
    
    assertEquals("30", frame.table.getValueAt(1,1));
}

Discussion

What I've found is that GUI testing works "ok" for some things but there tends not to be an easy way to do other things. For example, I didn't see an easy way to programmatically click "Enter" in the edited box. (I know you can set up a Robot and fool with events, but I wasn't willing to try that hard.) Another example is the difficulty of dealing with pop-up menus, tooltips, dialogs, etc.

The other problem I have is that I just don't think of everything important without actually seeing the GUI grow. So even if I use test-first, I'll mix it with running the application and seeing how it looks. (For example, we didn't address column widths, or seriously look at what happens when you select multiple cells, and my solution doesn't leave the grid element selected after an edit.)

I try to keep the GUI work limited to direct setup and hookup of objects. It can be very tempting to say "oh, this is GUI so I don't need to test it," but don't over-do that. Make sure that the GUI is not making application-level or algorithmic decisions.

I do sometimes end up creating programmatic acceptance tests, like the last test. I try to minimize my need for these because they're a fair bit of work to program, and the customer can't really look at them and verify them. In this case, the test checks the hookup, which is ok. But for other spreadsheet tests, I might automate something that reads an input file and a "golden" result, then compares to the actual contents of the sheet. In this way I could check formulas, circularity, etc. without having to hand-code so much.


Part 6, Bonus Round

This part is designed to let you assess the flexibility of your design. I'm just going to provide general directions; you can develop your own tests for any area you undertake.

"Obvious" extensions

There are a lot of places where we can expand on our spreadsheet. How much impact would each of these have? Do they affect the core of what you've done?

  • Allow spaces in formulas (if you don't already).
  • Support for floating point numbers.
  • More cell data types: date, money, etc. (Do you have different cell types yet?)
  • Range specifications and simple functions. For example, "=@sum(b1:c29)" should add up those cells. (Does this affect circularity detection?)
  • GUI improvements
    • Color index column a different column and right-justify it.
    • Select by row or column.
    • Column widths default to a useful size.
  • Input/output. Allow your spreadsheet to read and write "csv" (comma-separated value) files.
  • Add a graphing module.

Performance

If you haven't performance-tuned, your solution probably re-parses (or at least re-evaluates) a value every time it's referenced. Try a test like this:

A1 1
  A2 =A1+A1+A1+A1+A1+A1+A1+A1+A1
  A3 =A2+A2+A2+A2+A2+A2+A2+A2+A2
  A4 =A3+A3+A3+A3+A3+A3+A3+A3+A3
  A5 =A4+A4+A4+A4+A4+A4+A4+A4+A4
  A6 =A5+A5+A5+A5+A5+A5+A5+A5+A5
  A7 =A6+A6+A6+A6+A6+A6+A6+A6+A6
    etc. as needed

  Then change A1 to 2.

Get a profiler and see where the time is being spent. Can you speed up evaluation? How hard would it be to cache values during an update cycle (instead of re-calculating for each reference)? Would it make sense to have cells know which cells to notify when they changed? What other tests would help you understand the spreadsheet's performance?

Non-rectangular spreadsheet

The last couple challenges focused on using our Sheet as a traditional 2-d grid. But the idea of dependency and automatic updating is not really dependent on the shape.

How about a tree-based spreadsheet? Instead of A1, C32, etc., let the cells be named as words separated by dots: "Sales.Y2000.Q1" and so on. You might have formulas like this:

Sales =Sales.Y2000+Sales.Y2001
  Sales.Y2000 =Sales.Y2000.H1 + Sales.Y2000.H2
  Sales.Y2000.H1 =43000
  Sales.Y2000.H2 =54000

  Sales.Y2001 =Sales.Y2001.H1 + Sales.Y2001.H2
  Sales.Y2000.H1 =45000
  Sales.Y2000.H2 =74000

Updating should still work as before; if you increased Sales.Y2000.H2 by 6000, then Sales.Y2001 and Sales should increase by that same amount. You might like to display all this in some sort of tree control rather than a table.

  • How much of your GUI code could you retain? (If it's very much, it could be a sign that you're doing things not strongly related to the GUI; or perhaps your code is just so generalized it doesn't care about the shape.)
  • How big a change is this to your parser?
  • How much of the Sheet object could remain the same?
  • If you'd known you were moving to a tree in the first place, would you have built the same Sheet, or a different object?
  • If you did the performance tunings, did they make it easier or harder to make this change of direction?

Another approach might be to decouple the sheet from any pre-conceived shape. Perhaps you'd have a user interface that let people drag-and-drop formula pieces and connect them with arrows (like some programming environments or a data flow graph).

  • Would the Sheet object still be helpful?
  • Could you easily evolve from the Sheet to something more directly useful?
  • To the extent your design needed to change, would you feel test-first let you down?

Discussion

I don't expect miracles out of test-first design, but I expect that I'll get a reasonably robust solution, that feels like it ended up a little different than it would have from an up-front design.

I hope the experience of "test-first without writing tests" has inspired you to try "test-first by writing tests".

Again, thanks to all participants. I've really enjoyed corresponding with people, reading code, and seeing the discussions about this exercise.


Closing Observations

Here's a few observations about the test-first challenge, from the author's point of view.

  • It feels "different" to do this as a remote customer/test-writer (compared to how it feels to do it as a programmer), but I hope it still gives a fair flavor of the process.
  • One set of tests definitely does not generate one solution. Some aspects have been similar between solutions, but others have varied a lot. The two biggest areas of variation: parsers and circularity. In parsers, I saw operator precedence, recursive descent, and several ad- hoc approaches. For circular references, I saw a counting approach and a couple variations of "track whether a reference is to a partially evaluated node".
  • Someone commented that "what's simplest depends on experience," and that's my experience too. For example, the ad-hoc parsers seem simple in one sense, but I have a lot more confidence in the correctness of the "standard" compiler-type solutions, and they didn't seem all that complex.
  • I find myself writing two types of tests. (I'm trying some new vocabulary here…) "Generative tests" are the ones that push my design forward, moving in broad strokes to a solution. "Elaborative tests" are the ones I write to really nail down a feature, sort of a theme and variations approach. The "challenge" tests tended toward the former. (For example, you can't look at my tests and tell where and whether spaces are acceptable in formulas, and you'd need more tests to really verify the expression processing.)
  • I hope you were able to create the extra tests needed to check your implementation.
  • The solution and the test influence each other. I believe if I'd been working towards an ad-hoc parser (rather than recursive descent, say), I'd have come up with different tests. I tend to do enough design that I can say "this could work", and then let the tests take it where they will. Like the old song, I "hold on loosely" to the design, and if the tests lead elsewhere, that's ok.
  • If you get the chance, it's really worth tackling the same problem two or three times. My test-first skills have improved with practice. Next time I try the spreadsheet, I might ignore the math part, and just have text cells and formulas with a "," operator that appends strings. Or try the same thing again, but have in mind "use operator precedence, and handle circularity by counting depth" instead of "recursive descent and partial evaluation check".

Participant Solutions

Several people who took the challenge in January 2002 have posted their solutions. I encourage that you not look at these before you attempt your own solution, and I encourage that you do look at them after you finish.

If you've posted your solution and want to share it, let me know and I'll add you to this list.

Resources and Related Articles

[Written January, 2002; fixed missing setup in part 3, February, 2003. Updated Sept. 04 to mention GUI test tools. Added James Cakalic's solution, August, '05.]

Acceptance Test Mechanisms

An acceptance test is a test that the user defines, to tell whether the system as a whole works the way the user expects. Ideally, the acceptance tests are defined before the code that implements the feature.

Acceptance tests are run frequently in an XP project, usually daily or more often, and certainly every week. The tests give a picture of how much of the desired functionality will work. Because the tests are run so often, there is a payoff if they can be automated.

Progress on acceptance tests is one of the crucial metrics that XP projects often collect.

There are several mechanisms that can be used to run acceptance tests:

  • Manual Test
  • GUI testing tools
  • Code
  • Script
  • Spreadsheet
  • Template

Manual Test

The simplest mechanism for running a test is to just make a plan on paper, and then do the steps manually.

GUI Testing Tools

There are commercial tools that let you run a system, recording your activities and the system's responses. You run it once to establish a "golden" copy, and then automatically after that to verify that things haven't changed.

This might be useful for some parts of testing, but it's not as good as you might think.

  • If the interface changes, the tests are invalidated. (The tools' exact approach determines how significant an interface change will trigger this problem.)
  • The interface is often (usually?) not the core of the problem being solved. Thus it might be the case that interface is done toward the end of the project. All of the functionality should be tested along the way.

Code

In the code approach, you get a programmer to write the code that will run the tests you specify. This code is usually straightforward. Some teams use frameworks such as JUnit to manage and run the acceptance test code.

Script

"Script" is a simplified form of code. Programming languages typically have several ways to repeat actions, to do something if a condition is or isn't true, and so on, but most test code doesn't need these more complicated parts of programming.

So, programmers might create a scripting language. Tests are usually stylized, simple code, and the programmers will provide a way to run them. The language might be a subset of the one the programmers are using (e.g., Java), or a more "scripting" language like Perl, Ruby, or Python.

Example:

doc("Extreme Programming Explained", "Kent Beck", 1999);

doc("Planning Extreme Programming", "Kent Beck and Martin Fowler", 2000); expect("Extreme and Programming", 2); expect("Explored", 1); expect("Fish", 0); expect("Explored or Planning", 2);

 

 

 

Downsides: Script languages still require a lot of attention to details, and may have unwanted complications. Even in this simple example, there are several issues that the test writer has to be careful about:

 

  • Don't forget the semicolon at the end of each line.
  • How do you put quotes inside quotes?
  • Numbers vs. strings

 

Benefits: It's not all bad news:

 

  • It's easy to get started (by adapting an existing language)
  • Gives the user direct ownership of the tests
  • The solution is flexible.

 

Providing a scripting language is often the easiest way to get a user writing tests.

 

Spreadsheet

 

When you get to their essence, many tests can be represented in a spreadsheet. This gives the user a well-understood way to create, edit, and manage tests.

 

Example:

 

Title Author Year
Extreme Programming Explored Kent Beck 1999
Planning Extreme Programming Kent Beck and Martin Fowler 2000

 

 

 

Query Expected Comment
Extreme and Programming 2 AND found
Explored 1 No booleans
Fish 0 Term not present
Explored or Planning 2 OR found

 

Spreadsheet formats are not hard for a program to handle. Spreadsheets almost always have a way to save files in a format where commas, tabs, or "|" bars separate the fields. Then the program just reads one row per line and does what is needed. Different types of tests might use different programs, but still work with the spreadsheet approach.

 

Template

 

As you grow a bunch of different tests, it becomes more of a pain to write a new small program for each test type. You can use a "template" approach that combines scripting and spreadsheets.

 

The user writes the script that would test one row, and then "templatizes" it by replacing data values with a generic name. Then one program can create the actual script by merging the template and the spreadsheet.

 

Example:

 

*{
test($Comment);
expect($Query, #Expected);
}*

In this made-up example template, the part contained in the "*{ }*" brackets is repeated once per spreadsheet row.

Conclusion

There are several options for creating and running acceptance tests. Each team will have to decide which option (or combination) will work best for it.

Resources and Related Articles

[May 15, 2001.]

Interface Seams: Rethinking the Spreadsheet

Spreadsheets have traditionally been based on a rectangular grid. We’ll look into simplifying formulas, and then at data structures other than a matrix.

Spreadsheets often have columns specified by alphabetic labels, and numbered rows. A specific cell can be identified by its coordinates, e.g., B17. Similarly, rectangles can be specified by their corners, e.g., A2:D14. Formulas are stored in cells in the table. The benefit of an electronic spreadsheet is that formula cells are automatically updated when the values they depend on are changed. A downside of the "formula in cell" approach is that the formulas themselves are mostly invisible.

We’ll examine a number of alternative spreadsheet structures, with an eye toward keeping the formulas explicit and visible.

The simplest "spreadsheet" has no cells. The formula has no value to work with – so in effect it’s a constant.
 

constant

The next simplest spreadsheet has one data value and one formula (which takes a single argument).
 

data value
formula

A list of items provides an interesting level of complexity:
 

value1
value2
value3
value4
formula over list

Commutative operators are particular useful: if
            (a op b) op c = a op (b op c)
then we can drop the parentheses and treat the operator as an n-ary operator. Such operators include addition and multiplication, but also "average". (This is reminiscent of the "reduce" operator in APL: take a list and an operator, and the formula is formed by inserting the operator between each value.) Our formulas can be very compact if they’re just describing one of these expressions. For example, "+" may describe adding all values in a list.

Trees provide a way to structure formulas explicitly. The interior cells provide a place to "hang" the operators. We can treat each node as having a list of children (thus reducing it to the previous "list" example).

For example, we may want a sum of products:

    +
38
   
  / | \  
*
2
   *
30
   *
6
/\
1 2
  /\
5 6
  /\
3 2

Notice how the leaves of the tree are the data values, and the interior nodes have operators. The value at the interior node is the result of applying the operator to the list of children.

If we’re willing to restrict our formulas somewhat, we can require that each node at a given level have the same operator. (The example tree has this property.) We can have a very compact formula that describes each level. For our example, "+*" describes a sum of products.

What can we bring back to the rectangular spreadsheet? The idea of a visible formula, working across a row or down a column.
 

  Test Test 2 Score  
Obs 1 1 2 5 10 * 27
+
Overall
Obs 2 3 2 7 12 +
Obs 3 -1 0 6   5 +
  10 
+
Total
 

 
(We’ve set it up so the formula cells contain the resulting value, the operator that applies, and an optional label.) By letting operators "stretch" across the cells to which they apply, we’ve removed the need for specifying cell and rectangle coordinates.

In summary, we’ve used a variety of alternative spreadsheets to explore some variations, and brought a new formula style to the rectangular spreadsheet. The new style retains the table structure and dynamic updating that makes spreadsheets so responsive, but also separates the data from the formulas in a way that will make their hidden contents more explicit.