Forgotten Failures

Abraham Wald was a Hungarian Jew who fought in WWII and was responsible for untold numbers of pilots and crew members coming back alive from engagements over Europe. He did all this from an office in New York City. His weapons were mathematics and statistics, and his battlefield was data used to create solutions to real battlefield problems.

When American planes came back from missions, they were riddled with bullet holes. They gathered data on where the holes were and then asked the Statistical Research Group (SRG), where Wald spent much of WWII, to help them optimize where to armor the planes. Armor is heavy and caused the planes to use more fuel and be less maneuverable, so it had to be used sparingly. You can get away with less armor if you concentrate it on the areas of greatest need, so they asked Wald and his team to analyze the data and recommend where to put it.

The data showed that the planes were hit most often in the fuselage, the wings, and the tail. Wald looked at the data and realized something was missing. The data was from planes that had returned. So where were the bullet holes on the planes that didn’t return? He surmised that the planes that failed to make it home were hit in places where the planes that did make it weren’t hit. The armor needed to go where the bullet holes weren’t—the engines.

This is a textbook example of Survivorship Bias, a common error in logic that creates a distortion in how we see the world. It occurs when we focus on the success stories and do not consider failures. We look at the college dropout turned billionaire and falsely assume that anyone who works hard enough and has a good enough idea can succeed. In doing so, we simply forget that for every success there are tens of thousands of failures.

Forgetting about the failures leads us to ignore what statisticians call “the base rate,” the probability of a given result. If you play roulette, you can expect to win one out of 38 games, or 2.63%. That is the base rate. If you mistake winning as the rule instead of the rare exception that it is, you will lose a lot of money playing.

Survivorship bias also messes with how we understand cause and effect. Stories of people who beat the odds lead us to see correlation where there is mere coincidence. Most businesses fail. Most people do not become wealthy or famous. Most risks lead to loss. Does this mean we should not try? No. It means we should be realistic.

There is no easy way to overcome survivorship bias. I would love to give you “Three Easy Steps to Success in Overcoming Survivorship Bias,” but by now you understand that this would almost certainly be an oxymoron. Overcoming any cognitive bias takes hard work and honest self-reflection. This is true for individuals and equally true for organizations.

A good place to start is to practice asking ourselves what is missing from the data we are seeing. Are we excluding things we don’t like, things we fear, things that are not popular? Like Wald, we need to be open to a solution set that is found in the holes in our data (see what I did there?)

Another thing we can do is to memorialize our failures and research the failures of others. If we are launching a product, let’s find out why other product launches (ours or other’s) failed. If we are buying another company, let’s study the most statistically relevant reasons acquisitions fail. If we are planning a change in our processes, let’s study examples of change management ideas that didn’t create buy-in and effect change.

Failure is far more likely than success, so we should not be ashamed to fail, but we should be ruthless about learning from our failures. Like Wald, we must not look at just what we can see. We must consider all the things that started on the same path but didn’t make it.

I am not opposed to optimism. I believe in our vision, and I believe we will overcome the unforeseen obstacles and succeed. I must, or I wouldn’t keep leading. If I take my responsibility as a leader seriously, I am compelled to be honest about the “base rate” and not forget the failures, ours and others, that have so much to teach us. I agree with the stoics in this ideal which is best summed up in this quote by Marcus Aurelius, “You have power over your mind, not outside events. Realize this and you will find strength.” Let’s not forget failure as we strive for success and embody The Kimray Way.