And I want to start that with a very Simple Model of performance
where you can think of performance in terms of Real Tendency + Luck.
And we've been talking about this a little bit, we can formalize it and
don't get too put off by the baby math here, but in formal terms you
can think of performance y as a function of x true ability and
e some error, some randomly distributed error around 0.
Now, what does that mean for when we sample on extreme performance?
What underlies extreme success and failure?
If that's the model of the world, and everything we've been saying so
far says it is, that there's some noise in these performance measures,
what does it mean when we sample on extreme performance?
Well, it means that extreme success
suggests that the person might in fact have superior ability or
tried very hard, but also that they got lucky, that error was positive.
And conversely, extreme failure perhaps means inferior ability or
that they did not try very hard but also negative error or that they got unlucky.
We can be sure that as we sample very extremely on performance measure,
a noisy performance measure, and they're all noisy,
we can be sure that when we go to the extremes, we get extreme error as well.
What are the consequences of that?
There's one very important consequence, and that is in subsequent periods,
error won't be negative again, it will regress to the mean.
You'd expect it to be zero.
Error is, by definition, zero.
And if you got very positive error in one period,
you would expect less error in the following period.
This is a notion called regression to the mean, and
it's one of the most important notions in performance evaluation.
An example.
There was a study a few years ago of mutual fund performance, in the 1990s.
The study divided the 1990s into two halves.
1990-94, and then 1995-1999.
And they looked at the top 10 performing funds from the first half of the decade.
And here, I'll show them to you.
We anonymized them.
This is just supposed funds A through J, and their performance in the early 1990s.
There were 283 funds in this study.
These were only the top 10 performing funds.
Then they did two things.
They go and ask how do these funds perform in subsequent years?
And they did an interesting thing in between.
They ask people, what do they predict happened in the next few years?
What do they think performance would be realized in the second half of the decade?
Here are the predictions, the estimations from the people that they ask.
They thought the top performing,
they didn't think the top performing firm A would again be the top performing firm,
but they thought maybe tenth and so on down the list.
E which is the fifth performing firm they thought, well, maybe 44th and so
you can see that they didn't expected the firms to be as good, but
they expected some regression to the mean.
Then they looked to what actually happened.
What actually happened?
It ranged from 129th, 21st, 54th.
The interesting this is that on average, the firms performed.
Their rank was 142.5.
What is the significance of 142.5?
It's half of the total number of firms in the study.
In other words, the average performance of the top 10 firms, in the second period,
the second half of the 90s, was perfectly average for this sample.
They've regressed entirely.
The top 10 mutual funds in the top half the 90s and
of early 90s regressed entirely to the mean in the second half of the 90s.
If that's the case ,what does that say about how much skill versus luck
was involved with how those firms did in the first half of the 90s?
If they regress all the way to the mean in the second period,
it suggests that there was no skill.
That the differences that we saw, and
there are huge consequences to those differences because we know that new
funds flow to successful funds, were in fact, entirely based on luck.
There are many other examples.
Danny Conoman, Nobel Prize winner Danny Conoman gives a famous example of being
an officer in the Israeli Air Force.