Ace of Spades HQ

November 30, 2005

Expert Political Judgment
— Dr. Reo Symes

The New Yorker does a good rundown on Expert Political Judgment: How Good Is It? How Can We Know?, a new book by Berkeley academic, Philip Tetlock.

As the title suggests, its all about how accurate the predictions the political and economic pundits we see and read actually turn out to be. And Tetlock isnt just some crank, shooting off his mouth. Hes pretty rigorous:

[H]is conclusions are based on a long-term study that he began twenty years ago. He picked two hundred and eighty-four people who made their living commenting or offering advice on political and economic trends, and he started asking them to assess the probability that various things would or would not come to pass, both in the areas of the world in which they specialized and in areas about which they were not expert. By the end of the study, in 2003, the experts had made 82,361 forecasts

And his conculsion? The experts are full of it.

eople who make prediction their businesspeople who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtablesare no better than the rest of us.

Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable.

And the big shots are the worst:

Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an experts predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote.

Tetlock points to a number of explanations, most centering on the psychological foibles inherent in assessing uncertainty, along with simple stuff like the fact theres really no puditocracy penalty for getting it wrong.

Anyway, the piece is relatively short, not one of those New Yorker book/articles and well worth the read.

(N.b. My predictions were not part of the study, and remain, of course, rock solid certainties. You can continue to bet the farm on the Good Doctor's forecasts)

(h/t Jane Galt)

Posted by: Dr. Reo Symes at 08:00 AM | Comments (11)
Post contains 416 words, total size 3 kb.

1 Yeah, duh. Most bloggers seem to spend their days pointing out how wrong these experts are. The inverse relationship between predictions and accuracies is obvious for most liberal "experts" as they are merely giving voice to their dreams. The wronger they are, the more air they get. That way, Chris Matthews can circle jerk with fellow Bush bashers.

Coulter pointed this out in Slander, how CNN threatened to replay a clip of her being wrong about how long the impeachment trial would last. Gee, who won that battle?

I mean, if you are going to offer your credibility up to pose WP as a WMD, then you have to expect to never be right. Even if WP=WMD will get you NYT cred. BTW, does the WP=WMD mean that Iraq had WMDs? So Bush was right all along!

Posted by: joeindc44 at November 30, 2005 08:19 AM (8TcCs)

2 I predicted those results.

Posted by: polynikes at November 30, 2005 08:34 AM (m2CN7)

3 My guess is that Tradesports is a far better predictor of political events than the prognostications of puffed-up, self important pundits.

I wish there would have been some way to bet on whether a military draft would be installed during the Bush presidency. I would have laid at least a grand against it, and I'm certain not one lefty would have taken the bet. Even they know how full of shit they are.

Posted by: The Warden at November 30, 2005 08:45 AM (Zxtyv)

4 He probably has a whole chapter devoted to Paul Krugman and Jim Cramer.

Posted by: Moonbat_One at November 30, 2005 09:23 AM (ivB4r)

5 Or Dick Morris - he seems like the classic case of someone who's rewarded more for being entertaining than being right. He's a smart guy, but his bad predictions could fill a small library. The latest I remember (besides Hillary v. Condi) was that Bloomberg v. Ferrer would be down to the wire.

Posted by: Yaron at November 30, 2005 10:07 AM (rj3dl)

6 I predict the nation is about to be swept by Kusinich fever!! Has anyone seen Dennis' new bride? She's like six foot two. And he's like, five foot nothing, one hundred and nothing.

Like that scene in Deuce Bigalo. "That's a huge bitch!"

Posted by: fugazi at November 30, 2005 10:57 AM (KyUPY)

7 The journal Science had a study on experts several years ago that came to the same conclusion. It focused on experts advising judges on whether a given criminal was likely to commit more crimes if released. Needless to say, they were flipping coins.

The author of the study said that one thing surprised him. He expected that judges actually believed the experts. In fact, he found that judges recognized perfectly well, that the experts were blowing smoke. They went with the expert recommendations because they had nothing better, and it shifted the responsibility off them and on to the "experts."

Posted by: Bob Hawkins at December 01, 2005 09:51 AM (eZ0vq)

8 Reminds me of the bio of Jimmy the Greek I saw on ESPN yesterday. He didn't know shit about football, but he sure could call them.

Posted by: Mark V. at December 01, 2005 01:02 PM (/653L)

9 ment is very suitable designer handbagsdesigner bagsDiscount designer handbags

Posted by: adsds at January 24, 2010 03:09 AM (bYUmd)

10 I agree that Dr. Reo is Expert in what he does.

SEO Pakistan

Posted by: SEO Pakistan at November 01, 2011 04:08 AM (Q7nl0)

11 "Expert Political Judgment” showed that political and economic "experts” were no better at predicting world events than flipping a coin would have been.
jacob hiller

Posted by: davidtan at November 03, 2011 08:32 AM (E0PR6)

Hide Comments | Add Comment