Here is a sample question for prospective financial, business or supply chain analysts, “how many piano tuners are there in Chicago?”
This question comes from the business best seller Superforecasting: The Art and Science of Prediction. Before I give you the “answer” to the question, let me tell you a bit about this book’s genesis.
While one of the authors has been doing research on forecasting for decades, the more recent origin of this book stems from a colossal failure of the intelligence community. The community had predicted with absolute certainty, no room for error, that Saddam Hussein had weapons of mass destruction. That was a $1.7 trillion mistake for the U.S.
Subsequently, the Intelligence Advanced Research Projects Activity (IARPA) agency sponsored research to improve how the intelligence community forecast global political and economic trends. IARPA created a forecasting tournament in which teams would compete to generate accurate forecasts on all sorts of tough economic and political questions. One team, the author’s team, was called the Good Judgement Project (GJP). “Each team would effectively be its own research project, free to improvise whatever methods it thought would work, but required to submit forecasts at 9 a.m. Eastern Standard Time every day from September 2011 to June 2015. After two years, the GJP was doing so much better than the other teams, that the competitors were dropped.
At the conclusion of the trial, it was reported that GJP forecasts were 30 percent more accurate than the intelligence community. In short, a small team of “superforecasters” – most of them retired professionals that participated in the project as an interesting hobby, outperformed the multi-billion dollar intelligence community’s 20,000 analysts. Further, they had no access to classified information!
So what is the “answer” to the question of how many piano tuners there are in Chicago? Assuming you can’t look on Google or in the Yellow pages, how would you go about answering this? The accuracy of the answer is not as important as the thought process the job candidate exhibits. It is also the thought process of the superforecasters in the book.
The key is to begin to break down the question and try to make reasonable estimates for different components of the question.
You could “nail” this question if you “knew four facts:
- The number of pianos in Chicago
- How often pianos are tuned each year
- How long it takes to tune a piano
- How many hours a year the average piano tuner works”
These components are then broken down as well. You might not know the answer to any of these questions, but making estimates on these components is better than just pulling a number out of the air.
“1. How many pianos are there in Chicago? I have no idea. But just as I broke down the first question, I can break this down by asking what I would need to know in order to answer it.
- How many people are there in Chicago? I’m not sure, but I do know Chicago is the third-largest American city after New York and Los Angeles. And I think LA has 4 million people or so. That’s helpful. To narrow this down … (set) a confidence interval – a range that you are 90% sure contains the right answer. So I’m pretty sure Chicago has more than, say, 1.5 million people. And I’m pretty sure it has fewer than 3.5 million people. But what is the correct answer within this range? I’m not sure. So I’ll take the midpoint and guess that Chicago has 2.5 million people.”
- What percentage of people own a piano? Pianos are too expensive for most families – and most who can afford one don’t really want one. So I’ll put at one in one hundred. That’s mostly a black-box guess but it’s the best I can do.
- How many institutions – schools, concert halls, bars – own pianos? Again I don’t know. I’ll again make a black-box guess and say that it’s enough to double the per person number of pianos to roughly two in one hundred.
- With these guesses, I can do some simple math and conclude that there are fifty thousand pianos in Chicago.”
If you are interested in how the authors broke up the other three components to the question and answered them, I’d suggest buying the book. I found it to be a quick and engrossing read.
I won’t tell you the final answer. As I said before, it really doesn’t matter that much. It is the process, the way a candidate’s mind works that is important.
