Kahneman and the real meaning of Man Utd’s 6-1 defeat

As a Man Utd fan since the days of George Best (I am just about that old – I precociously started supporting them aged 3 in 1973), I was of course pretty disappointed by the events at Old Trafford on Sunday: BBC match report on Man Utd 1-6 Man City. But more interesting to me were the reactions of the football experts afterwards. While the media were gurgling about the match, I was also reading Nobel laureate Daniel Kahneman‘s article in the New York Times on the over-confidence of experts: Daniel Kahneman – Don\’t Blink! The Hazards of Confidence. It struck me that many of the football experts on Radio 5Live, Sky Sports and elsewhere were making a lot of “cognitive fallacies”. Kahneman has some clues as to why.

Their claims: City showed themselves superior to United, they thrashed them and taught them a football lesson. The problem: recent history actually points to these occasional heavy defeats being more galvanising for Ferguson United teams than the teams who beat them – see today’s Independent (at the end of the Balotelli story): United stand: how they react to humblings. United lost similarly badly to their nearest rivals in three recent seasons and went on to win the league. Add to that, that United played with 10 men for much of the second half – a fact mentioned but not given much significance in most reports – and you wonder why so many pundits are now talking about United in such dire terms?

Kahneman describes how, as a young conscript in the Psychology Branch of the Israeli army, he had helped evaluate candidates for officer training, by watching their performances in team exercises. He duly made recommendations: so and so is a natural leader, so and so is slow to take on others’ ideas etc. The problem was, when these candidates got to officer training school, the evaluations turned out to be useless predictors of how well they would do there:”Our forecasts were better than blind guesses, but not by much.” Did the young Kahneman then change his approach? Here’s the interesting thing: he didn’t. Unable to take on board the scale of the re-think required, he ploughed on as before: “The statistical evidence of our failure should have shaken our confidence in our judgments of particular candidates, but it did not. It should also have caused us to moderate our predictions, but it did not.”

What allowed him to feel OK about carrying on like this was that his system of evaluating performance had “the illusion of validity”. There was visual evidence he could point to of what each candidate had done in the exercise (working as a team to get a log over a wall); he could reference the results and who had played what role in achieving them. The judgments were valid, in their own terms. It’s just that they bore little relation to what the Israeli army actually needed to know. But it was easier to stick with a familiar process.

The 6-1 scoreline gives the illusion of validity to a story of United on the wane. As Football 365 put it, “it feels like City have arrived to overtake United as the team of the present and that their time has come.” But it’s October, United are in contention at 2nd in the league and have played some scintillating football, on a level above recent seasons. They had been favourites to beat Man City on Sunday. So why are many experts ignoring the bigger picture and projecting so dubiously from one disastrous match?

Kahneman calls his own failure an example of “WYSIATI” (what you see is all there is):”People who face a difficult question [SR: like the probable outcome of the Premiership title race] often answer an easier one instead, without realising it [SR: like what was the score in that match].” Kahneman goes on to say: “Confidence is a feeling, one determined mostly by the coherence of the story and by the ease with which it comes to mind, even when the evidence for the story is sparse and unreliable. The bias towards coherence favours overconfidence.”

Football is especially prone to a cognitive fallacy we all fall prey to – football writers often tell the story of the final result, not the story of the play. Football analysis in this country is remarkably free of undeserved victories, because the analysts post-rationalise the events of the game to make themselves come across, reasonably enough, as rational and wise. In explaining why the victory happened, the pundit often gives the impression it was always likely to happen as it did. The uncertainties felt at the time are forgotten. A goal is taken to mean more than just a goal, it’s an expression of dominance. Yet even the best teams concede goals and even lose matches by big scores. We struggle to explain this; it jars. So we pretend it doesn’t happen (unless you’re The Independent).

There’s a qualitative research lesson in all this too: beware the convenient and easy story. The best qualitative research has to look at all the surrounding evidence too, including contradictory data. The best qual researchers tell stories well but only after going through a quite different process: being very tough on themselves and taking on board inconvenient truths. You have to doubt your own judgment every step of the way. Even mine.

Of course it could be that United do fall apart; who knows? But do seasoned Premiership watchers seriously think this result heralds anything more than what it was – a one-off defeat, albeit a terrible one? Or were they just getting carried away with the moment?