Superforecasting is a super entertaining and informative story with a noble agenda it’s not subtle about.
- Forecasts are important - we invaded Iraq because of high-certainty predictions they had WMDs, and we were wrong. Obama famously said it’s a 50-50 coin flip that Osama’s in that house.
- Forecasting is hard. Accurate forecasts are the result of synthesizing many different viewpoints in one head, while also recognizing and adjusting for our personal biases. Making useful predictions means being wrong a lot - which is why news pundits stick to vague predictions that can be stretched to cover any length of time or degree of change. Furthermore, improving your forecasting ability is hard. To improve, you need to take a step back and review previous forecasts, but it can be especially tempting to call it lucky/unlucky, or to succumb to hindsight bias, when you say “oh, I definitely gave it a 60% chance of occurring”, when in reality you gave it 20% at the time.
- That said, forecasting is learnable, and you don’t have to be a genius to be a good forecaster. “It’s not what superforecasters think, it’s how.” The Good Judgement Project asked thousands of people weekly questions about hard problems - “Will there be a marine casualty in the Korean Peninsula by the end of 2013?”, and used the Brier Score to rank individual predicting ability. The ones who consistently outperformed the rest of the crowd by a wide margin were labelled superforecasters, and Tetlock lays out their thinking strategies and personality traits.
One strategy that stuck with me was the “outside view, then inside view”. Say I tell you about a family in Ohio with two children and two parents working full-time, with a stay-home grandparent. The children like playing outside, and they have a big yard. What’s the probability that this household has at least one dog? You might piece together a story about this particular family in which the parents buy a dog to keep the children busy while they’re at work. The dog loves their big yard, and the grandparent keeps an eye out in case the kids forget to feed it. So you say, 90% chance. This is the inside view first; the outside view would be to step back and ask “What percentage of households in the US have a dog?” That answer, 58%, would give you a baseline anchor, and then you’d dig into the details of this family to adjust that percentage incrementally. Or perhaps you instead ask “What percentage of households with two children have a dog?” or “What percentage of houses (as opposed to apartments) have a dog?” This strategy yields way more accurate predictions. I read this book because a data scientist at Twitch got everyone a copy. I personally have put these lessons to use making better work-time estimates. Given a hiccup that might delay a release, I’ll either adjust the completion date if there’s no hard deadline, or set a probability that I won’t be able to account for the hiccup in time for this release, and go from there. The more relevant information I can consider when making these estimates, the better. Superforecaster predictions are to the percentage - while we mostly make round number predictions like 40/60%, superforecasters weigh new information with more granularity - they frequently think “this is between a 1% and 10% change in confidence… hmmm, probably less than 5% though, so I’ll go with 3%”. I’m currently at a 85% chance I’ll be code complete by the expected date for mobile broadcasting (6% some shitty Samsung devices require special error handling, 4% chance network switching logic will need to change when we field test, 5% ingest edge cases will take longer than expected). There’s a lot more discussion about mental attitudes of superforecasters, and this book even presents and considers its counter-arguments.