Tuesday, August 7, 2012

Survivorship Bias

A new fitness guru in the making!
If one puts an infinite number of monkeys in front of (strongly built) typewriters, and lets them clap away, there is a certainty that one of them would come out with an exact version of the Iliad. Upon examination, this may be less interesting a concept than it appears at first: Such probability is ridiculously low. But let us carry the reasoning one step beyond. Now that we have found that hero among monkeys, would any reader invest his life's savings on a bet that the monkey would write the Odyssey next?
In this thought experiment, it is the second step that is interesting. How much can past performance (here the typing of the Iliad) be relevant in forecasting future performance? The same applies to any decision based on past performance, merely relying on the attributes of the past time series. Think about the monkey showing up at your door with his impressive past performance. Hey, he wrote the Iliad.
The major problem with inference in general is that those whose profession is to derive conclusions from data often fall into the trap faster and more confidently than others. The more data we have, the more likely we are to drown in it...
...we tend to mistake one realization among all possible random histories as the most representative one, forgetting that there may be others. In a nutshell, the survivorship bias implies that the highest performing realization will be the most visible. Why? Because the losers do not show up. 
Fooled By Randomness, Nassim Nicholas Taleb, pp. 135-146

One needs look no further than fitness-reality-TV, strength and fitness gurus, and even the "training template of the month", to see this "survivorship bias" play itself out. On the surface, contestants, clients, and athletes make dramatic progress. Gurus, systems, and templates parade their success stories as proof of their efficacy as trainers and coaches. We want to believe that their athletes are successful because of a combination of the coach's wisdom and the athletes' hard work - sometimes this may be true. It is important, however, to understand that what we see are only the "survivors" - those that fail to prosper are not in the spotlight. For every "I ADDED 100lbs TO MY SQUAT IN 8 WEEKS!" or "I Lost 50lbs in 2 Weeks!" there may be dozens of failures, aborted attempts, and even injuries left in the wake.
Yes, there are many qualified and competent coaches and trainers out there. There are also many barely competent enough to hold a clipboard and shout encouragement, let alone design a training program. Honestly, a monkey in a golf polo shirt could do as well much of the time. The question is, why do we keep flocking to monkeys hoping for the Illiad? And, how many of us are willing put our money and bodies on the line, betting on the monkey writing the Odyssey?

No comments: