How likely is "likely"?

Dear reader, Sherman Kent, originally a Yale University history professor, spent 17 years in the CIA during World War II and the Cold War-era, where he pioneered many of the modern methods of intelligence analysis. He is today often described as “the father of intelligence analysis”, and oversaw over 1500 National Intelligence Estimates: the US Intelligence Community’s authoritative assessments on a particular national security issue.

At the start of his career, during the Korean War, a report was published stating that a Soviet attack on Yugoslavia within the year was a “serious possibility”. Kent was puzzled as to what, exactly, “serious possibility” meant. He personally interpreted it as meaning that the chance of attack was 65% – but when he asked other analysts and policymakers what they thought, he heard estimates ranging from 20% to 80%.

Why, then, do people use vague words to describe probabilistic outcomes? Phil Tetlock, who wrote the book Superforecasting, believes that it is because “vague verbiage gives you political safety”. The old joke about economists is that they have “successfully predicted 10 of the last 3 recessions”. Paul Krugman is famous for predicting diametrically opposite outcomes at different times, and then going back to highlight the one prediction that came true – and conveniently forgetting about the rest.

More experienced “superforecasters” are better at calibrating their certainty levels, and perhaps most importantly, are better at understanding the bounds of their knowledge, and when they need large confidence intervals.

The best way to improve your forecasting accuracy is to accumulate a large body of experience. Importantly, you need to get feedback as to the results of your forecasting: keeping track of historical decisions and bets.

Remember to include both probability and magnitude. It’s not enough to put a % probability on something - you need to price the outcome. For example, knowing that a coin flip bet has a 50/50 chance of happening doesn’t tell you anything as to whether or not you should take the bet. You need to know the payoffs for each scenario.

It is thus important to know the difference between Risk and Uncertainty. Risk is loosely used in everyday speech, but properly defined, it refers to the probability of something happening. Mathematically, it is well-defined. There is a 50/50 risk of a coin landing heads or tails. You can calculate exactly the risk of something happening.

Uncertainty, also known as “Knightian uncertainty”, refers to the fact that you do not know the probability distribution in the first place. For example, if I say that I have a jar with both red and black balls in it, but I don’t know how many of each are inside, there is uncertainty as to whether I will draw a red or a black ball. If I knew exactly how many of each ball were inside, I would be able to calculate the risk of drawing each ball.

Many problems in forecasting, such as black swan events, happen because what you think is risk is actually uncertainty.

Returning to the issue at the start, that of linking qualitative words to quantitative probabilities, it is ideal to also be explicit and use numbers when describing probabilities and magnitudes.

A more recent Harvard Business Review study showed that while most people have roughly the same probability beliefs, not all do. In particular, there is a difference between men and woman: women use uncertain words and phrases more often than men do, even when they are just as confident.

Furthermore, it is important to avoid culturally biased words, such as “slam dunk”. While that meaning might be obvious in one particular culture, its understanding can vary widely in different contexts.

From the chart below, you can see that while each term has a “hump” around which most people agree, some terms have long “tails”. The terms “real possibility” and “might happen” are great examples of just how widely meanings can vary.

PS. Sherman Kent’s efforts to quantify what were basically qualitative judgments did not prevail within the CIA.

Yours Sincerely,
Henry Chong