What brands do you think of when you think about political polls? For most people, the names Gallup, Mason-Dixon, Rasmussen, Pew and ,perhaps, Zogby come to mind. Smart analysts and political junkies are quickly adding the name Silver to that list.
Nate Silver is the founder of FiveThirtyEight.com, a website that aggregates poll results and presents a consolidated view to users. But 538 goes beyond simple aggregation. The site analyzes the past results of each of the pollsters and scores them according to their bias. Every poll will have a bias - what Silver refers to as Pollster-Introduced Error (or PIE). PIE focuses on the errors introduced by the methodology of the pollster, as opposed to those purely a result of the statistical sample size. Some lean a bit left and others a bit right. One key to understanding differences in poll results is to understand the bias
of each of the polls.
In developing their aggregated forecast, Five Thirty Eight assigns each poll a weighting based upon the pollster's historical track record, the sample size of the poll and the recency of the poll. They apply other statistical techniques to reduce the impact of outliers and refreshes polls that may be dated. Then, using all of this analysis, they run 10,000 simulations to provide a probabilistic assessment of electoral outcomes.
So, how well does all that number crunching work? In the presidential race, Five Thirty Eight's final pre-election day projection was that Barack Obama would win 52.3% of the popular vote, while John McCain would take 46.2%. The final tally: Obama 52.6%, McCain 46.1%. The only state that 538 called incorrectly was Indiana, where they projected McCain to win a highly competitive race; Obama, in the end, won Indiana by nine-tenths of a percent (49.9 to 49.0%).
Where does Nate Silver get his insights into political polling? Is he a Gallup veteran? Has he been toiling behind the scenes at Congressional Quarterly? Hardly. Prior to launching FiveThirtyEight.com, Silver was a writer and analyst at Baseball Prospectus.com, which does similar statistical analysis to project likely statistics for baseball players. In other words, he's just a stats geek.
There are other poll aggregators on the web, such as pollster.com and realclearpolitics.com. But while each does a good job, over the past few months as I closely followed the presidential election as well as various Senate and House races, I found myself turning to FiveThirtyEight for trustworthy data and analysis. And I wasn't the only one. As we moved into November, FiveThirtyEight was among the top 2,000 visited sites according to Alexa. In addition to great data, FiveThirtyEight prides itself on its transparency and provides a detailed breakdown of the six-step process that serves as the basis of their methodology.
As content becomes commoditized, its increasingly difficult for aggregators to differentiate their offering. Nate Silver and FiveThirtyEight.com are a great example of how to add significant value to commodity data through analytics and a great user interface. FiveThirtyEight also exemplifies how new media builds on the efforts of the traditional media. The underlying data for their analyses comes from those traditional pollsters mentioned in the opening paragraph of this post.
As we move on from the 2008 elections it will be interesting to see what five thirty eight does to keep their site relevant. Or, perhaps Nate (and his colleague Sean) will move on to another venue. Perhaps they'll shift their statistical models to looking at the financial markets or into determining the potential future movement of energy costs. But in the meantime, 538 is not yet done with 2008. They are still analyzing the results of contested Senate races in Alaska, Minnesota and Georgia and have already begun to handicap the races for the 2010 open Senate seats.
UPDATE: The New York Times did a nice profile piece today on Nate Silver and 538.
