Hard numbers

Numbers are tricky, especially when you want to use them to understand business problems. A book I read a few years ago and several blog posts more recently have highlighted this for me, so I got the urge to write about the challenges inherent in understanding with numbers and a couple of helpful tips I’ve picked up from the Agile community.

In The Elegant Solution, Matthew E. May devotes a chapter to talking about the importance of picking the right things to measure for your business (Chapter 11, “Run the Numbers”). The Elegant Solution is one of my two favorite business books (together with The Seven-Day Weekend by Ricardo Semler), filled with Lean principles learned from May’s years teaching for Toyota. But while most of the book is quite practical, its practicality kind of unravels in this chapter.

The problem is, the first several examples he gives show just how darned hard it is to get numbers right. It’s hard enough to decide what you want to measure. But the examples go way further, showing how even math PhDs can get correlations and probabilities horribly wrong. The author doesn’t even seem to be trying to make the point of how hard it is to get the math right. He seems to be trying to merely make the point that it’s important to get the math right. It just turns out that every example is more about getting the math wrong.

The first examples he gives aren’t even from business, starting with the Monty Hall problem. This problem evokes heated debates, with very smart people giving different answers and standing by those different answers insistently. The vast majority of people get it wrong, including people with advanced study in mathematics. I got it wrong at first, and it took several tries to finally accept the right answer. (In my defense, I did figure it out before getting out of college!)

In another example, May arguably gets it wrong himself:

You’re playing a video game that gives you a choice: Fight the alien superwarrior or three human soldiers in a row. The game informs you that your probability of defeating the alien superwarrior is 1 in 7. The probability of defeating a human soldier is 1 in 2. What do you do? Most people would fight the human soldiers. It seems to make intuitive sense. The odds seem to be in your favor. But they’re not. Your probability of winning three battles in a row would be 1/2 X 1/2 X 1/2, or 1/8. You have a better shot at beating the alien superwarrior. It’s a simple problem of probability. [Matthew E. May, The Elegant Solution, page 158]

Here’s the thing: If the chance of beating each human soldier is purely random and independent of beating any other human soldier, then the probability of beating three in a row is indeed 1/8. But if there’s any skill involved in the game, then the chances are not independent. When I’m playing a video game, I typically will grow through levels of skill where certain kinds of opponents become easy to defeat. When I start playing the game, maybe I almost never defeat “human soldiers”, but after some time playing I get to a level of skill where I almost always defeat them. So if I can defeat the first soldier, I will likely defeat all three; if the probability of defeating the first soldier is 1/2, the probably of defeating all three is arguably nearly 1/2 as well, which is clearly better than the 1/7 chance of beating the “alien super-warrior”.

I was reminded of this chapter from The Elegant Solution when I saw a flurry of blog posts on a similar problem recently: See Jeff Atwood’s question, his own answer, Paul Buchheit’s response, and the discussion on Hacker News… The problem is just as simple as the Monty Hall problem, and the response it just as heated. Paul Buchheit points out that the simple English statement of the problem can be parsed two different ways, which result in two completely different answers (both of which I verified myself by Monte Carlo simulation!). In another realm, Semyon Dukach suggests that the current financial crisis is due precisely to the difficulty of numerical intuition.

The examples of success with numbers in The Elegant Solution all come down to identifying very simple metrics underlying the businesses in question. Jim Collins gives similar counsel in Good To Great. But the problems mentioned here show that even given a very simple mathematical statement, you can still get into lots of trouble. In The Goal, Eli Goldratt gives some fascinating advice about how to orient your business toward the true goal of business (I won’t spoil the plot by saying what “the goal” is; it really is worth reading the book). But here again, the whole story line of The Goal shows just how unintuitive those principles are, and how long and painful (though valuable) a process it can be to learn them through real-world experience.

So what do we do? A couple pieces advice I’ve picked up over the years might help:

1. Measure, don’t guess. Specify the problem precisely enough to implement a Monte Carlo simulation, and then run it several times. This is the only way I’ve been able to convince myself of the answers to tricky problems like the Monty Hall problem. The more I think about the problem, the less sure I get about the answer. This principle also applies to optimization of code: Start with the simplest thing that could possibly work, and then measure how it performs under realistic conditions. The thing that needs optimization is often not what I guessed it might be.

2. Measure “up”. Mary and Tom Poppendieck talk about this principle in detail in their Lean Software Development books (the phrase is mentioned on page 40 of Implementing Lean Software Development). It is an antidote to local optimization: The more you focus on localized metrics, the more confused you’ll get. As they write, “The solution is to … raise the measurement one level and decrease the number of measurements. Find a higher-level measurement that will drive the right results for the lower-level metrics and establish a basis for making trade-offs.”

What other advice do you keep in mind when numbers get tricky?

Clicky Web Analytics