Survivorship Bias
3 min read

Survivorship Bias

We often draw conclusions from a sample group without pausing to consider whether the sample is a true reflection of what we should be measuring.

What is the Model?

Survivorship Bias suggests that we humans are drawn to consider only successful outcomes and this can lead us to overestimate the chances of our own successes.

The easiest way to describe this bias is by paraphrasing a story -- unfortunately I can't remember where I read this initially but it's certainly not my own.

The Returning Bombers Fallacy

Supposedly, the story goes, some very smart designer was asked during the middle of World War II to design a better bomber for the Air Force. The Air Force presented him with their ideas, including photographs of the bombers that had returned from air raids over Germany. The photos showed the damage caused by the German anti-aircraft defenses. The Air Force told the smart designer that whatever new aircraft he came up with would have to have additional armour in those spots because clearly that's where their aircraft keep getting hit.

The smart designer saw that information for what it was, though, and recognised the Air Force's Survivorship Bias. There would be no need for extra armour in those spots, he told them, because the photographs proved the current aircraft could get shot in those areas and still make it home safely. Rather, all the spots without damage need the extra armour, as none of the aircraft shot there had made it back home at all.

The Air Force was using 'returned aircraft' as their sample group when really they should have used 'departing aircraft'.  Had they introduced the extra amour the Air Force wanted, they would have confidently sent their new planes into the sky still just as vulnerable as before.

How Does the Model Apply to Corporate Counsel?

The Returning Bombers Fallacy is a helpful way to describe Survivorship Bias, but we shouldn't take it too far.

For example, when a Commercial Representative comes to you as Corporate Counsel and asks for a contract that adequately protects against a well known issue, the correct answer isn't to respond smugly with 'Ah yes, but what about all the other problems you aren't even aware of?'. Obviously, the right response is to draft the agreement as best as possible with the information at hand.

But Survivorship Bias is pervasive and Corporate Counsel can add value by protecting decision makers against their natural exuberance.

For example, when a Business Development Executive suggests the Company enter a new market on the back of a competitor's success, the skeptical Corporate Counsel can add value by noting the potential for Survivorship Bias. Is there perhaps a need for further due diligence – has the Executive considered all others that sought to enter the market and failed?

The goal is not to rain on any parade but rather to ensure an accurate understanding of the Company's chances for success. The Corporate Counsel might be able to structure the venture in a way that avoids those problems others fell to or, after looking into the venture further, the Corporate Counsel might identify regulatory risks that others avoided by sheer luck. Again, the goal is not to prevent the Company's entry into the new market but to ensure it's done with care and on a fully informed basis.

Another area where Survivorship Bias might come up is in negotiations where the other side refers a track record of performance or to 'market standards'. We can do a separate post on why references to 'market standards' are of limited use generally, but they can be especially misleading when combined with Survivorship Bias.

For example, when a buyer states that they will not pay more than $XYZ for a business acquisition because that's the standard multiple within the market, consider whether the 'market' being referred to is subject to Survivorship Bias. Houses sold at auction, for example, ignore those houses sold privately before the auction date.

Consider also when Outside Counsel refers to their excellent track record whether they might be tweaking their sample size some. It's a lot easier to win 90% of cases when you're able to settle the losers before they ever get to Court.

Look Out For

Any time someone looks to use a representative sample as a way of supporting their argument.

Remember, often the representative sample won't be described as '40% of [ABC] shows us that...' but rather 'Every customer we spoke to thought ...' or even 'We've done this plenty of times before and it has always worked out.'

Where to Read More

Survivorship Bias is covered in Rolf Dobelli's excellent book The Art of Thinking Clearly (affiliate link).

You can also read about Survivorship Bias at the Farnam Street Blog.

--

What do you think? Have I got this one right or have I missed the point? Do you have your own example of Survivorship Bias in a corporate setting? Let me know.