Testing Exceptions: Harder Than It Looks

Posted on by

Exceptions are one of the most complicated constructs in many languages. Because of that, testing exceptions is one of the harder tests to do.

Consider the following code. Assume block1 through block3 are straight-line code, each ending with a statement that could possibly throw an exception.

block0
try {
    block1
    block2
    block3
} catch (Exception e1) {
    handler1
} catch (Exception e2) {
    handler2
} finally {
    final cleanup
}

This code has a number of potential paths – how will you test them?

Paths

What are the paths through this code?

Here’s my visualization: (Straight lines are executed, dashed lines are not. Dots show which blocks are run, and squared dots throw exceptions.)

One happy path and nine error paths through the code

Let’s start with the happy path:

block0 – block1 – block2 – block3 – final

Now assume an exception occurs at the end of block1. There are three paths, depending on what error is thrown.

  1. block0 – block1 – handler1 – final
  2. block0 – block1 – handler2 – final
  3. block0 – block1 – final  [exception escapes to caller]

There are three paths when an exception occurs at the end of block2:

  1. block0 – block1 – block2 – handler1 – final
  2. block0 – block1 – block2 – handler2 – final
  3. block0 – block1 – block2 – final [exception escapes to caller]

Finally there are three paths when an exception occurs at the end of block3:

  1. block0 – block1 – block2 – block3 – handler1 – final
  2. block0 – block1 – block2 – block3 – handler2 – final
  3. block0 – block1 – block2 – block3 – final [exception escapes to caller]

If your handler itself has a try-catch, the combinations multiply. 

Transactional Thinking

In a databases, transactions either succeed or fail. I bring that mindset to the code I’m testing. I usually want to ensure that work is either completed or rolled back, not left in an intermediate state. I want to ensure that any resources (such as network connections or file handles) are freed. Such resources are usually finite:), so if you don’t release them, a series of exceptions could consume them all.

A Real Example (Somewhat Truncated)

Consider this code. What paths should you test? Does the code behave correctly?

try {
    fileReader = new FileReader(filename)
    reader = new BufferedReader(fileReader)
        :  // code that might throw
} catch (IOException e) {
    // whatever
} finally {
    try {
        if reader != null { reader.close(); }
        if fileReader != null { fileReader.close(); }
    } catch (IOException e) {
        // ignored
    }
}

The try code may have many statements that could cause an exception. This means that fully testing the try-catch is a challenge. To fully test it, you need to test each situation for the try clause (gets past the first potential exception, the second, etc. to the end). You’d need to test that each situation was handled properly, both by its catch and finally clauses. You’d also need to test all the paths through any containing try-catch statements, and that the ancestor methods on the call stack each handle any bubbled-up exceptions correctly. 

There’s a flaw in this code’s exception handling – do you see it?

What happens if closing the reader throws an exception? Then the fileReader never closes. In a finally clause that’s releasing resources, each of those requires its own try-catch.

Triggering Exceptions

Testing the happy path its (relatively) easy. Testing exception paths is harder.

The hard part is that the code inside the try clause must be made to throw. There are two basic approaches:

  1. Set up conditions so the real “try” body throws the desired exception.
  2. Make the “try” code call substitute code, a test double that doesn’t do real work but just throws an exception.

The “real code” approach is not ideal for unit testing. I use that approach only when I’m really testing the underlying mechanisms, e.g., if I’m the one writing the code for the open() method itself. Then I might do things like check a FileNotFound exception by attempting to open a file that doesn’t exist.

But usually, I’m the client of open(), and the test double is a better approach.

Why? Two reasons:

  1. It’s hard to trigger many exceptions! For example, “disk space full” can require filling a disk – what a pain, and likely to create problems for the rest of the system.
  2. Real exceptions are often slow. Anything involving files, networks, databases, etc. is certainly slow relative to a test double. (How long does a disk write take? How long would it take to fill an 8 TB disk with junk data to trigger that FileNotFound exception? How long would it take to tests a network call that does retries with a series of random delays?)

Test Doubles

Where does an exception come from? Either we throw it ourselves, or it comes from a method on another object.

There are three typical ways to make a replacement. I don’t have a definitive rule to choose between these; I do whatever seems easiest in context.

Lambda/Closure

Replace the function call that throws (or the whole try-catch) with a lambda expression. This pushes that code into the caller. To test, we create a lambda expression that just throws the exception.

Fake Object

Create an object that has the same method (or subclasses) the throwing object. In the fake object, make the target method throw an exception.

Mock Object

Use a mock object (typically from a mock object library) that can be instructed to throw an exception when a particular method is called.

Throwing an Exception in a Closure

Here’s the sketch of a refactoring to use the closure approach:

Assume we have code like this:

:
try {
  // main body that throws
} catch(es) / finally
  : 
}
:

A. Extract Method on the main body that throws. (It may need a number of parameters.)

B. Convert the direct call to a closure call:

extracted(x, y);    =>     var fn = extracted; fn(x,y);

C. Introduce Parameter on fn so the caller provides the function. (Default to having callers pass the name of the extracted method.)

D. A test can now pass a closure that is { throw SomeException(); }

You may find that the extracted method belongs on a different class, and you may have to make other adjustments.

Conclusion

Testing exceptions is hard: try-catch is a complicated construct, and there are usually many implicit paths. To do those tests, you need to trigger the exception, either with the real code or a test double. Finally, we looked at a refactoring to move code to use the lambda / closure approach.

References

“Database transaction”, Wikipedia. Retrieved 2021-12-22.

“Spooky Action at a Distance: Exceptions“, by Bill Wake. Retrieved 2022-02-02.