What is regression to the mean?

Regression to the mean is the tendency for an extreme result to be followed by a more ordinary one, because extremes involve a run of luck that does not repeat. Nothing pulls the number back; the luck simply runs out. The danger is that whatever was tried in between gets the credit for a recovery already on its way.

A hospital runs a new programme in its 50 worst performing wards, and six months later those wards have improved. The programme takes the credit, but the worst 50 were picked partly for genuine problems and partly for having had a rough six months, and rough patches end by themselves. Why not just measure before and after, which sounds like the obvious approach? Because selecting for extremes creates the illusion: pick a group precisely because it is at its worst and bad luck is at its peak in that group, leaving one direction to travel. The lesson it teaches is worse than the wasted budget. Praise a pilot after a superb landing and the next one is usually shakier, while shouting after a bad landing is usually followed by a better one, which makes criticism look effective and praise look useless. Daniel Kahneman found flight instructors convinced of exactly that. The fix is not clever statistics but a comparison group, so pick equally struggling wards, leave them alone, and count only the gap between the two.

The tallest man you know will probably have tall children who are, on average, a little shorter than he is, and his family is not shrinking. Being unusually tall takes tall parents plus a favourable roll of the genetic dice, and dice do not roll the same way twice. It runs the other way too, so very short parents tend to have children taller than themselves. Height drifts toward the middle because luck is part of every extreme.