A user can treat a product as a black box – “I like what it does, even though I don’t know how it works.” A product owner doesn’t have that luxury.
Friend 1 – That’s a big trip you’re planning – you should check the oil.
Friend 2 – No need – I have a gas car.
[Told to me by someone who says they were there.]
We’ll look at several real situations where a product owner caused, or suffered because of, product decisions. (Sorry, no companies are named:) Then we’ll explore some models that could have helped reveal the problems sooner.
Failover that Failed
The system was deployed, fully replicated, transactions doubly written, with databases in different time zones, etc. The system worked for a few months, but when the main database went down, failover didn’t work.
The team had never tested the failover or the second site. The failover failed because of a network misconfiguration, but when they tested belatedly, they found that even had that been right, the failover wouldn’t have worked.
A Couple Similar Stories
A small team (2-3 developers) spent a couple years developing a multi-threaded, distributed system. When they shipped, it collapsed almost immediately. Why? It had never been tested with multiple threads or across machines, and wasn’t written right. It took 9 months to repair and release again.
Another company trained their programmers from COBOL to C++. After a couple years’ development, they tried to deploy at scale. The system ran out of memory. More than half of the team didn’t understand that they had to track and free memory.
Seen a Few Times
Some systems have a security story, “Email a user their password if they’ve forgotten it.” Doing this means the password is stored in cleartext or a reversible encryption. This vulnerability has been understood for decades.
I’ve also seen systems that had a cleartext password, for use by admins, but not shared with users. This is still a security concern: if someone gets a copy of your database, they can try these passwords on other systems too.
Greedy for Features
I’ve seen this one several times, but one case stands out to me. The (evolving) team had been delivering for 10-15 years, but almost every time the team wanted to do some architectural work, the product owner vetoed it.
The architectural work was not “let’s rewrite”, but consisted of things like: replace a no-longer-supported library; upgrade the compiler and later change compilers; update to the latest operating system; reduce duplication.
It came to a head eventually. Mac OS updates stopped emulating older versions, so the system was left unable to run on new machines (that couldn’t run the old OS). By this time, the architectural work was far more expensive than if it had been done along the way.
Product Owners Can Build Models
Real systems are complex enough that you can’t fully specify or understand them from the outside alone. You need a model of the inside of the system (and possibly models for the way the team works too).
Most systems have (or can have) a high-level architectural picture. It may be work to understand, but the architecture influences what your system can do and how it can do it. Non-functional requirements or properties of the system seem particularly prone to this lack of understanding.
A simple model can suggest a lot of helpful questions. Here’s a model of a (distributed) system sending information to a user:
We see a connection – so we can ask: How reliable is the connection? How fast is it? How big are the messages we’re sending across? Does the connection allow for real-time delivery (e.g., video)? Have we tested with the user’s connection (not just one inside our own network)?
Here’s another model: an over-burdened team:
(A plain arrow indicates a positive influence, an arrow with a circle a negative influence.)
Here’s another team model, reflecting the impact of never addressing architecture:
(The = on a line indicates its effect is felt after a delay.)
Here’s a simple model of an at-risk system:
Does it make sense to add a new room if the foundation is crumbling and termites are eating the walls?
Is It the Product Owner’s Fault?
Blame is not the right model: everyone involved is in a (social) system that drives behaviors. Most people do what they can, without bad intent.
Product Owners have to make difficult tradeoffs: there may not even be a right answer.
But in each of the cases I described, the whole team made a mistake, including the product owner. Even in the more technical cases, the product owner could have understood the architecture well enough to question the proposed solution or a lack of early testing.
A product owner is not in in a position to understand everything about a system. Consider the log4j problem of a few years ago: a logging tool used by thousands and thousands of teams turned out to have a security bug that would make any system that used it vulnerable to hackers. Programs of any size embody many, many decisions and have many dependencies. It’s too much for anyone to understand it all.
A 10,000-foot view is feasible for any system, and this can help you understand many challenges.
Conclusions
We briefly looked at a number of systems that failed for ostensibly technical reasons, but a product owner is in the right position to question a team’s approach and understand many important consequences.
There are many models a product owner can build to help understand the team dynamics and a team’s product.