Agile ’06 Workshop: Example-Based Specifications

This workshop was designed to explore Example-Based Specifications. Hosted by Brian Marick and Bill Wake.

Schedule:

  • 1 hour – One person plays customer, the rest of the team writes tests
  • 30 minute debrief
  • 1 hour – Small group discussions
  • 30 minutes – sharing and conclusions

Test Writing

Group 1: Domain – a statistical tool that evaluates inputs according to a formula, and plots the results.

Their test:

(A, B, and Y are inputs; b0, b1, b2, and b12 are outputs, produced by a process of matrix algebra.)

A B Y b0 b1 b2 b12
10 75 8 11.0 5.0 8.0 12.0
20 75 12
10 100 10
20 100 15

This is a textual representation of the graph

b0 b1 b2 b12 B-1 B-2 B+1 B+2
0 5 8 12 (1,1,RGB) (10,10,RGB) (10,1,RGB) (1,10,RGB)

Their expected graph (described here, though the team drew a picture):

  • a large X with each line in a different colors
  • labels +1 and -1 in the graph
  • a horizontal axis labeled “A” (also having labels -1 to +1)
  • a vertical axis (“Y”)
  • a graph name (“B”).

Group 2: Domain – a session scheduler, matching audiences to rooms

Test: Schedule a session –

Preconditions: user has logged in, room size < audience size

  • TU35 has iaudience 100
  • Room NICA has capacity 25
  • TU35 is not scheduled
  • NICA is available Monday 11-12:30

Execution:

  • User assigns TU35 to NICA at Monday 11-12:30
  • System displays a warning, “Room size mismatch… Continue or cancel?”
  • User selects “Continue”

Expected Results:

  • TU35 shows up scheduled in NICA, Monday 11-12:30

Group 3: Domain – a graphical tool for tracking and identifying criminal conspiracies. (Tool helps build a network of connections between people).

Their test:

Select template criminal
Create node
Create node
Select link template ?
Create link
Check count links 1
Delete node 1
Check count links 0
Create node policeman
Set first name Tom
Set last name Nassau
Create criminal
Set last name Soze
Select link template arrest
Create link Nassau to Soze
Check link exists
Select template policeman
Create node
Add role
Select node 1
Check has role policeman
Check has role drug dealer


Group 4: Domain – an expense approval manager

A GUI mockup:

    Manager approval:
    Purpose:

Type Date Description Amount Needs Receipt
    American Airlines SEA to MSP $600.00  
       

X

         

Issues: editing, are you sure?, manager approval, expense group

Test:

Type Date Vendor v Description Amount
Air tix 7/16/06 AA ORD to MSP 763.42

    Purpose: Trip to agile conference
    Manager approval:    Pointy-haired boss

An analysis of states:

New – Open, can’t pay, can’t approve, can’t close
Submitted – Open, can’t pay, can approve, can’t close
 

  Is Open Can Pay Can Approve Can Close
New        
Submitted        
Approved        
Paid        
Closed        

 


Group 5: A stock trading program

A screen mockup with annotations:

Start Time [   ] v : [   ] v 9:30-16:00 ET (market start to market end)
End time [   ] v : [   ] v
Stock Symbol [    ] 1-4 alpha
# Shares [    ] 1
Order Size [    ] 100->1 million (int) (+- 100)
  [Buy/Sell]  
Price $ [    ] (optional) numeric + 2 decimal optional
     
  [OK]    [Cancel]  

Test of algorithm:

  • Start time >= now >= 9:30 EST
  • End time > start time <= 16:00 EST
  • Distribution list (file) exists
  • Test invalid symbol entry
  • Test for valid symbol entry
  • Test all shares are traded if no limit price
  • Test trades don’t violate limit price
  • Buys or sells based on selection

Group 6: Insurance tracking program

Test for story “Insurer adds provider”:

Test 1 – Sunny day

Add table:

Johnson, David name 15ch, 15ch
1200 Nicolette Dr addr1 40ch
12345 addr2 5d or 5d-4d
123456789 taxid 9 ch num
ALLINA network id must exist in db

Read: (same)

Test 2- Invalid zip code (and more like this…)

Johnson, David name 15ch, 15ch
1200 Nicolette Dr addr1 40ch
1234 addr2 5d or 5d-4d
123456789 taxid 9 ch num
ALLINA network id must exist in db

Expected error: “Invalid zip could should be 5 digits”

Test for story “Analyst approves pending claims < 60 days since claim made”

Test 1 – Sunny day

Populate database – set up claim 1

Date of claim 30 days ago
Member name Doe, Jane
Provider name Johnson, David
Diagnosis 769
Charge $500.00
Network id ALLINA

Step 1. Analyst views pending claims < 60 days (display claim A => OK)

Step 2. Analyst approves pending…

    Signal “ok” = submit
    Claim disappears from list
    Message “approved”

Step 3. Read claim, status “approved”


Group 7: Domain: Shipping company.

Test: 1. Customer ABC call to know where shipment with order #33 is. The system should answer, “Last stop was Tampa, FL.”

 

Note Order # from Customer Request Answer from System
Truck left origin #33 last stop? Tampa, FL
Not in truck #34 last stop? nothing
Truck at destination #35 last stop? Daytona, FL
Truck at destination #35 arrived? yes
Not in truck #34 arrived? no
Not arrived #34 expected date? 26/7
Arrived #35 expected date? 25/7
Truck underway #33 arrived? no

Example context:

# Pick up origin Final drop destination Expected Date
#33 Tampa, FL Toronto, ONT 27/7
time 24/7 8 AM  
#34 Vancouver, BC Toronto, ONT 26/7
time  
#35 Toronto, ONT Daytona, FL 25/7
time 24/7 17:00 time 25/7 10:00

Test 2: I want the system to help me minimize empty truck displacement. For example, I want to be able to ask if there is an empty truck in Ontario on July 27. Arrival within 2 days.

Empty truck in? Shipment order  
Ontario/27/07 #33, #34 truck at city
Ontario/25/07   no truck because wrong date
Montreal/QC   no truck because wrong location

Debrief

We noticed these things from the experience of writing tests:

  • What to do is vague.
  • Developers tend to embellish.
  • Tests teach developers, but it’s a challenge to pick the right level.
  • It’s hard to turn descriptions into tests.
  • Tests (and collaboration) can help you discover new things.
  • The idea of a GUI became actions on a model (for the test).
  • Customers should come in prepared.
  • We need many questioners per answerer.
  • Someone came in late, and found that the examples helped them understand.

Topics

In small groups, we explored these topics:

  • Sufficient coverage: How do you know you’re done – what’s sufficient coverage?
  • Test styles: What is the relationship between example-based specifications and other styles of tests?
  • Product Owner: Techniques for interviewing product owners

Sufficient Coverage

  • When adding tests to a legacy system, how much do you “backfill”?
  • Should we just use “change detector” tests (record what the system currently does, knowing that it may change later, and may not even be correct).
  • Do we need all combinations?
  • Where do we fit example-based tests?

Test Styles

We suspected these things about example-based tests:

  • They may bring about exploratory testing, regression tests, better unit tests, acceptance tests. Load tests?
  • They provide a concise account of an edge case.
  • They serve to train new developers in a domain.
  • They provoke a certain style of conversation.
  • They may overwhelm developers with distracting detail (no “metaphor”).
  • Alternatively, developers may ignore examples! It happens…

Product Owner

  • Use Sophie’s Rule – keep asking “why?”
  • Ask the end goal, define the business problem, define acceptance criteria
  • Discuss requirements “as if you were blind” (without reference to UI)
  • Need a customer who is is readily accessible
  • Helps to talk to end users
  • Programmers need to understand the domain
  • Let product owners ramble

General Notes / What Next

  • Mind maps have utility in these conversations.
  • Examples are a style of conversation – the easiest kind to get.
  • “We’re going to practice” – “process miniature” exercises could help.
  • A whiteboard and pointing are powerful ways to focus attention.