Insights · Growth
Bayesian Thinking: How Every Test Should Make the Next One Smarter
For almost two years, nobody could find Air France 447. A new way of reading the failed searches found it in a week. The same thinking makes every customer test sharper than the last.
Most teams run a test, read the result and move on. It won or it lost, and whatever else it could have taught them ends there.
That is roughly how the search for Air France Flight 447 went. For a year, searchers looked where the plane should have been and found nothing. Then a team of statisticians treated every failed search as evidence instead of a dead end, and the next search found the wreck within a week (Stone et al., Statistical Science). The method is called Bayesian thinking, and it is the most useful habit I know for anyone who tests ideas on customers.
What is Bayesian thinking?
It is a way of updating what you believe as evidence comes in. You start with a prior, your best estimate before the new evidence. You ask how likely that evidence would be if your belief were true. Then you adjust, and the new belief becomes the starting point for the next round. Two habits sit at its core: you never start from zero, and you never stop learning.
How did Bayesian search find Air France 447?
On June 1, 2009, Air France 447 disappeared over the Atlantic on its way from Rio de Janeiro to Paris, with 228 passengers and crew aboard. Investigators knew the wreck lay within 40 nautical miles of the plane's last known position. Over the next year, four searches came up empty: ships and aircraft on the surface, a month of listening for the black box beacons, and two rounds of sonar on the seabed (Stone et al., Statistical Science).
In July 2010, France's air accident bureau asked the analysis firm Metron to start over. Metron built a probability map from what was known: the 40 mile circle, where planes came down in nine similar accidents, and where the drifting bodies pointed. Then it updated the map for every failed search, and allowed for the chance that the beacons had broken in the crash.
The 2011 search began near the center of that map. On April 3, 2011, after about a week, it found the wreck 14,000 feet down, inside an area the beacon search had already covered. A recovered beacon was later tested and gave no signal. Metron's team wrote that skipping a Bayesian approach in 2010 “delayed the discovery of the wreckage by up to one year” (Stone et al., Statistical Science).
Why do smart people get this wrong?
Usually because they ignore what they already know. In a 2007 training session, 160 gynecologists were given the numbers for breast cancer screening and asked what a positive mammogram means. Because the disease is rare to begin with, the best answer is that about 1 in 10 women who test positive has cancer. Only 21% chose it. Once they learned to turn the percentages into counts of women, 87% did (Gigerenzer et al.).
Business makes the same mistake from the other direction: we overrate our own ideas. At Microsoft, about two in three ideas tested in controlled experiments failed to improve the metric they were designed to improve (Kohavi, Microsoft).
| What happened | Number | Source |
|---|---|---|
| Searches for Air France 447 before the Bayesian map | 4, all empty | Stone et al., Statistical Science |
| Time to find the wreck once the search followed the map | About 1 week | Stone et al., Statistical Science |
| Gynecologists who chose the right meaning of a positive mammogram | 21% | Gigerenzer et al. |
| The same doctors after learning to use counts of women | 87% | Gigerenzer et al. |
| Ideas tested at Microsoft that improved the metric they targeted | About 1 in 3 | Kohavi, Microsoft; Bing |
When I put this to work
When I worked with a multinational brand, they had a loyalty problem. Their shoppers were strictly deal hunters, and deals alone don't build sustainable growth.
We didn't start with a guess. We ran focus groups, in person and online, to find out what would actually move the needle for these customers. That became our prior. Then we A/B tested what we heard:
- different kinds of notifications
- subject lines
- combo offers
- relevant, targeted information that was genuinely helpful, so customers could see the brand cared about them
Each result changed what we believed and shaped the next test, and the results got better as we went. Loyalty went up.
The focus groups gave us a starting belief. The tests told us how far to trust it. And like the car wash experiment, the lesson was that customers who feel valued come back for more than the deal.
How do you use Bayesian thinking on your next test?
- Write your prior down before the test. What do you expect, and how sure are you? Start humble.
- Decide what would change your mind before you see any data. Otherwise every result looks like proof.
- Count a loss as evidence. A flat test tells you where not to look next.
- Check the base rate before you trust a signal. A big jump in a tiny segment is often chance, the positive mammogram of marketing.
- Check your instruments. The beacons were broken. Make sure the tracking works before you blame the idea.
- Update often, in small steps. Good Judgment's top forecasters revised each forecast about 11 times, in small moves, and were 35.9% more accurate than other forecasters by the firm's own scoring (Good Judgment).
Don't ask whether the test worked. Ask what it changed about what you believe.
Read the full article
What is Bayes' theorem in plain English?
The chance your belief is true after new evidence depends on two things: how much you believed it before, and how much more likely the evidence is if the belief is true than if it is false. Counts make it easy to see. Take 1,000 women in a screening program. 10 have breast cancer, and 9 of them test positive. Of the 990 without cancer, about 89 test positive anyway. So of the 98 women with a positive test, only 9 have cancer, about 1 in 10 (Gigerenzer et al.).
Where does the idea come from?
It is named for Thomas Bayes, an English minister. His friend Richard Price found the essay among his papers and presented it to the Royal Society in 1763, two years after Bayes died (Royal Society).
Is a Bayesian A/B test different from a regular one?
They answer different questions. A classic A/B test reports a p value: how surprising your data would be if the two versions performed the same. The American Statistical Association stated in 2016 that a p value does not measure the probability that a hypothesis is true (American Statistical Association). A Bayesian test starts from a prior and reports the probability that one version beats the other. Either way, the bigger gain comes from treating tests as a series, where each result shapes what you test next.
What made the Air France 447 map work?
Metron blended three stories about the crash into its starting map and weighted them: an even spread inside the 40 mile circle (35%), where planes came down in nine similar accidents (35%), and where currents and winds said the bodies had drifted from (30%). The drift story got less weight because the currents were uncertain. Each failed search then lowered the odds in the areas it covered, scaled by how likely that search was to spot the wreck. The team capped its trust in any sensor at 90%, because in past searches, estimates based on manufacturer specifications ran optimistic (Stone et al., Statistical Science).
Metron had used the same approach in the searches for the submarine USS Scorpion and the steamship SS Central America, and it underpins the US Coast Guard's system for planning searches at sea (Stone et al., Statistical Science).
Where does Bayesian thinking go wrong?
With a bad prior. If you start out sure of something that isn't true, it takes a lot of evidence to move you. Metron's chief scientist, Lawrence Stone, says “a substantial amount of art” goes into the analysis (Communications of the ACM). The fix is to start humble, write the prior down where others can challenge it, and let the tests do the arguing.
FAQ
What is Bayesian thinking?
Bayesian thinking means updating what you believe in proportion to new evidence. You start with a prior, weigh what the evidence says, form a new belief, and use it as the starting point for the next round.
What is a prior in Bayesian reasoning?
A prior is your best estimate before new evidence arrives. It comes from what you already know, such as past data, base rates and similar cases.
How did Bayesian search find Air France 447?
The analysis firm Metron built a probability map from the known facts and updated it for four failed searches, allowing for broken beacons. The 2011 search started near the center of the map and found the wreck within about a week.
How many A/B tests actually win?
Fewer than most teams expect. At Microsoft, about one in three ideas tested in controlled experiments improved the metric it was designed to improve, according to Ronny Kohavi.
Is Bayesian A/B testing better than frequentist A/B testing?
They answer different questions. A p value says how surprising the data would be if nothing changed. A Bayesian result gives the probability that one version is better, given a prior. The bigger gain is treating tests as a series that keeps updating what you believe.
- Lawrence D. Stone, Colleen M. Keller, Thomas M. Kratzke and Johan P. Strumpfer, Search for the Wreckage of Air France Flight AF 447, Statistical Science 29(1), 69 to 80 (2014)
- Communications of the ACM, In Search of Bayesian Inference
- Gerd Gigerenzer, Wolfgang Gaissmaier, Elke Kurz-Milcke, Lisa M. Schwartz and Steven Woloshin, Helping Doctors and Patients Make Sense of Health Statistics, Psychological Science in the Public Interest 8(2), 53 to 96 (2007)
- Ronny Kohavi, Microsoft Experimentation Platform, tutorial on controlled experiments, KDD 2009 (slides)
- Bing Search Quality Insights, Large Scale Experimentation at Bing (2013)
- Christopher W. Karvetski, Superforecasters: A Decade of Stochastic Dominance, Good Judgment Inc (2021)
- Ronald L. Wasserstein and Nicole A. Lazar, The ASA Statement on p Values: Context, Process, and Purpose, The American Statistician 70(2), 129 to 133 (2016)
- Thomas Bayes and Richard Price, An Essay towards solving a Problem in the Doctrine of Chances, Philosophical Transactions of the Royal Society 53, 370 to 418 (1763)