Subjunctive Programming

Posted on by
Programming in the world of “What If?”

The subjunctive is the tense (or mood) used to talk about possible worlds. (You wouldn’t say, “I were in charge,” but with the subjunctive you would say, “If I were in charge.”) Subjunctive programming is an attitude and set of techniques that can help you obtain deeper insight into your problem and its solution.

Stance

The attitude you hold toward your software is crucial.

Some teams have legacy software, developed by people who left long ago, and occasionally maintained by others since then. In this situation, some teams’ mantra is “If it ain’t broke, don’t fix it.” Local repairs are the order of the day, even when the team knows this is papering over a deeper problem. This is a stance of fear.

We want a stance of courage. Our software should be under our control, pliable to our will. Our view is, “We have permission to experiment. We have a commitment to explore, but not a commitment to any particular result.”

Sometimes a bet doesn’t pay off; an experiment just isn’t an improvement. But often enough to pay back all, we gain a crucial insight that transforms our program.

Passing all the tests is important. Reducing duplication is important. Communicating through the code is important. But we seek insight as well. Our code may communicate what we’ve learned so far, but we have more to learn.

Ward Cunningham spoke at XP Universe ’01 of the feeling of “waiting for insight.” There is a niggling tension when we realize there’s a new insight to be had, but we don’t yet know what it is. We can savor this tension, and gently try things, all the more to appreciate the release when the new ideas work.

Environment

Two types of environment are involved: the social environment and the technical environment. As is so often true, the social environment has the bigger influence.

The social environment must be one in which it’s okay to fail in small ways. A lack of failure suggests that there is no risk-taking, that a team is chasing incremental sure things and avoiding riskier big wins. Some experiments won’t work out. This can be particularly challenging for management that is used to a tightly controlled task list.

I have a friend who is lead consultant on a large (non-XP) project that’s in danger of delivering late. Every time he stands up, the manager asks, “What’s the matter? Aren’t you working on such-and-such a task? It’s supposed to be done tomorrow.” How much experimentation will happen in an environment like this? The social context and reward structure discourage new ideas. This is plan-driven development at its worst.

When it’s socially acceptable to experiment, it helps to have a supportive technical environment. The foremost need is for a configuration management (CM) system. An experiment is usually done in a sandbox, a special area where changes can be made without accidentally affecting (or being affected by) other developers’ work. Some CM systems have a branch mechanism where you can check in intermediate versions without releasing them into the main line of development until you’re ready. (Many experiments won’t require checkins during the experiment, but it’s handy when you need it.)

In normal development, it’s XP practice to check in to the main line very frequently. Experiments should not plan to do this until they’ve been assessed. (Remember our stance – committing to an experiment doesn’t commit you to its result.) Checking in experiments to the main line mid-stream will blur the line between the main path of development and the experiment, and make it hard to back out the the experiment when necessary.

(I usually find it worthwhile to go the other way around – frequently pulling in changes from the mainline to the sandbox during the experiment. This makes it much easier to integrate when the experiment works out.)

Three Techniques

Thought Experiments

Some experiments don’t involve code; they involve thinking about the system.

For example, today your production system sees a maximum of 10 concurrent users. The peak CPU usage is 8%, peak I/O activity is 5%, and network use is negligible. How many peak concurrent users can we expect to be able to support?

A simple analysis might say, “CPU is the limiting resource, 100%/8% = 12.5, so we expect to support up to 12.5*10=125 users.” A slightly deeper analysis might say, “We have a rule of thumb that we should limit apps to 80% of capacity, so 80%/8% = 10 => up to 100 concurrent users.”

We still might need to do experiments to verify this, but already we have expectations. If the experiment showed a peak of 15 or of 500 users, we’d know we ought to investigate more deeply.

We can use “what-if” questions as well: What if we generated keys instead of using human-created ones? What if we were to make this class a Composite? What if we had 100 times as much data? What if everybody in the country had an account?

Thought experiments have limits, of course. Sometimes our mental models are just plain wrong, or are inadequate for the current situation. Other times, we reach impasses that can’t be resolved without looking at the real world. It’s important to know when to move from thought experiments to real ones.

Kent Beck uses a rule, “No technical discussion should go longer than ten minutes without running a concrete experiment.” This helps prevent daydreamy thought experiments that become endless blue-sky or gripe sessions.

Spikes

A spike is a code experiment focused on a narrow aspect of a problem. It may involve playing with the program we’re developing, writing a new small program, or trying something else.

For example, you might be worried about database performance in a crucial area of your application. You might try a spike where you run the slowest query (on random data) as many times as possible in 15 minutes. The reality is that your users will do a mix of many query types, but this way you get a worst case without as much work as if you tried to model a realistic workload. (You may run another spike to verify that caching effects aren’t making you look too good.)

Or, perhaps you’d like to switch to a new logging subsystem. If you’ve never used it before, you might write a small application that logs a lot but doesn’t do any real work.

A spike lets you quickly explore an abstraction of your real problem, without getting caught up in irrelevant details..

Refactoring

A lot of discussion about refactoring focuses on smells, transformations, and improving code. These are important concerns. But refactoring can be an exploratory tool as well. By using refactoring’s safe transformations, we can find out the real consequences of a proposal (rather than guessing, and cleaning up all the bugs later).

Refactoring lets us play with the balance of responsibility between objects, and lets us introduce new abstractions when we need to.

For example, consider software that has an underlying model with multiple views on it. (This is an abstraction of a real situation.)

The Compound has a list of View1 objects. A View1 will only be in the Compound if it was explicitly added to it. Given a Compound (and other information not shown), the ViewFinder can find all the View2 and View3 objects that have the same base. When the ViewFinder was originally introduced, View2 and View3 were not conceived as views on the Base, but they had evolved to become that.

We decided to try an experiment: what if the Compound was regarded as the set of all views implied by what was added (thus eliminating the ViewFinder)? This turned out to simplify a number of things, so we kept the result.

We had valuable, working code; we didn’t want to jeopardize that. The code communicated what we understood, but we re-conceptualized this corner of code, and learned to understand even better.

Conclusion

The Scrum software development [Schwaber] process grew out of new product development ideas, and its leaders have always stressed knowledge creation as a key dimension of software development.

By actively trying variations on our software theme, and by learning to better sense the tension of missing insight, we can work toward better software – and better ideas.

Resources and Related Articles

[Written September, 2003.]