The World's Greatest Election Forecast, in the World

by: Chris Bowers

Tue Aug 03, 2010 at 16:00


At long last, I have finally completed all of the research for my 2010 Senate (and, eventually, Governor) forecast methodology.  I am extremely pleased with the results, which I believe show it to be the most accurate methodology produced anywhere.  The methodology is explained below.

But first, here are today's numbers!

Senate Picture, August 3rd, with Rasmussen
Most likely outcome: Democrats 52 seats, Republicans 47 seats, Charlie Crist 1 seat

Of the 100 Senate seats, 86 are either not up for re-election, or have a polling average where one party has a 100% chance of victory (if the election were held today).  Among those 86 seats, there are 48 Democrats, and 38 Republicans.  Here is a chart featuring the other 14 campaigns:

Senate picture, competitive campaigns chart, August 3rd, with Rasmussen

The 48 currently safe Democrats, plus the 4.05 wins projected in these 14 campaigns, comes out to 52.05 Democrats, or 52 seats.  Charlie Crist is also projected to win one seat.

Senate Picture, August 3rd, without Rasmussen
Most likely outcome: Democrats 54 seats, Republicans 45 seats, Charlie Crist 1 seat

Senate picture, competitive campaigns chart, August 3rd, without Rasmussen

The 48 currently safe Democrats, plus the 5.91 wins projected in these 14 campaigns, comes out to 53.91 Democrats, or 54 seats.   Charlie Crist is also projected to win one seat.

Notes:

  • * = Has primary challenger, but heavy favorite
  • The "current Dem winning %" column projects the chance of Democratic victory if the election were held today.  It is not meant to predict the chance of the Democratic candidate winning in November.
  • Every Senate seat not listed here currently has either a 0% or a 100% chance of a Democratic victory.

Senate Forecast Methodology

I strongly believe this to be the most accurate statewide electoral forecasting methodology published anywhere.  Additionally, it is simple enough that almost anyone can reproduce it, no matter their level of background in statistics or polling.  This simplicity also means transparency, as almost anyone can both understand the assumptions I am making and check my arithmetic for accuracy.

The methodology is extremely simple: just take the simple mean of almost all polls that had the majority of their interviews conducted during the final 25 days of a campaign (see the notes below for more info).  That's it.  And it works, too:

Error rates, final predicted margin to final vote margin, 52 closest Presidential, Senatorial and Gubernatorial general election campaigns, 2008-2010
Pollster 538 Simple 25-day mean
Mean error 2.79 2.82 2.55
Median error 2.15 2.16 1.67
In the 52-closest Presidential, Senatorial and Gubernatorial campaigns where final margins were published by both 538 and Pollster.com, the simple 25-day mean resulted in significantly less error.  The 25-day simple mean had 9-10% less error on the mean, and 22-23% less error on the median.  Additionally, it was the most accurate in 21 of those campaigns, and the second most accurate in 19.

Further, the 2008-2010 performance of the 25-day simple mean was not a fluke.  Since 2004, across the 145 closest Presidential, Senatorial and Gubernatorial general election campaigns, its mean error rate has been 2.57, and its median error rate has been 1.76.  That is a consistently strong performance that will be difficult for any methodology to surpass, or even equal.  As of this writing, I know of no methodology that has done so.

Now, in the extended entry, answering some likely questions / objections:  

Chris Bowers :: The World's Greatest Election Forecast, in the World
So, what's that you say?

  1. But you can't average polls that have different methodologies! I have heard this claim as long as I have been blogging.  While it is nice deductive reasoning, it does not hold up to empirical research.  NCPP has found that the average "candidate error" (defined as half of the mean total error figures I produced above) for nearly 600 individual polls taken during the final eight days of the 2004-2008 general election was about 1.8.  By comparison, the average "candidate error" for the 25-day simple mean was about 1.3.  So, poll averaging was more accurate than individual polls, and by a significant amount.  Sure seems like you can average polls to me.

  2. But you are only telling us what was more accurate in the past, not what will be more accurate in the future! True! Just because it was more accurate in the past does not mean it will be more accurate than other methods in the future. Further, I fully expect that other electoral forecasters have conducted their own research, and improved their methodologies. I guess we will see who did their homework the best, and who made the correct assumptions, after November 2nd.

  3. What's up with the "current Dem win %" column? That is the odds of a Democratic victory in that Senate campaign, if the election were held today.  I am not projecting into the future.  It is calculated based on the error rates of the 144 campaigns looked at in the research I performed to produce this methodology.  For example:

    • Dem Candidate A trails by 2.25%. Given that 58 of the 144 campaigns have had error rates greater than 2.25%, and that the error has an even chance of breaking in favor of Democrats or Republicans, divide 29 by 144 to arrive at a 20% of Democratic victory in that campaign, if election were held today.

    • Dem Candidate B leads by 1.33%. Given that 91 of the 144 campaigns have had error rates greater than 1.33%, and that the error has an even chance of breaking in favor of Democrats or Republicans, divide 45.5 by 144, to arrive at a 32% of Republican victory in that campaign, and thus a 68% of Democratic victory, if the election were held today.

  4. But you don't weight polls by recent-ness! No, I don't.  And yet, my results are still more accurate.  In fact, I am pretty sure this is why they are more accurate, even if I can't prove that assumption.  I am pretty sure this methodology works because voter preferences really don't change much in the final stages of a campaign.  Without voter preference changing that much, adding more polls into the average and not weighting them produces more accurate results due to the central limit theorem.

    Basically, this methodology is about finding the sweet spot between two variables.  Adding more polls into the average improves accuracy, while including older polls reduces it.  It appears that 25-days from an election is the "sweet spot" that results in the least total error when these two variables are combined.

  5. But you don't weight polls by sample size! Nope. And yet my results are more accurate.

  6. But you don't weight polls by past accuracy! Nope. And yet my results are more accurate.

  7. But you don't weight polls by house effect!  Nope. And yet my results are more accurate.  However, I am open to weighting by house effect, if a test showed it was more accurate to weight by house effect.  I just couldn't find a comprehensive list of the house effect for every pollster to conduct this test.

  8. Why do you include partisan polls internal campaign polls, and multiple polls from single polling firms?  I do so is because I tested the total error of the averages both with and without all three of those variables. The averages had less error when all three of those types of polls were included.

  9. But you are only looking at campaigns within 18.50%! I am just not interested in forecasting blowout elections. Looking at campaigns within 18.5% or less still allows me to look at all targeted Senate and Governor campaigns, as well as the most expansive definition of "swing state" possible.  Go any further out, and we are just not looking at competitive elections.

  10. But you are focusing on the margin, rather than on the candidate raw totals!  Elections are not academic exercises--this is about knowing who is winning and who is losing.  If another forecaster can come closer to predicting the final raw number for each candidate, good for him or her.  However, I want to know how close elections are, who is going to win, and who is going to lose.  In the end, that is all that matters.

  11. Why do you produce two forecasts, one with Rasmussen and one without?  Showmanship, mainly. I imagine readers are interested in seeing a forecast without Rasmussen polling. The forecast with Rasmussen polls remains the "official" forecast.

    That said, the scandal at Strategic Vision, along with Rasmussen's born-again Republican house effect and their new outside funding, does make me suspicious.  So, I want to see what the numbers look like without Rasmussen, too.

That's it.  As always, I am very interested to read your comments.

Methodological notes

  • The data used in these calculations can be found here. (zipped folder with six Excel files)

  • I tested the performance of the simple mean from 2004-2010 for several other cut-off dates (10-day mean, 15-day mean, 20-day mean, 21-day mean, 24-day mean and 30-day mean).  The 25-day mean was chosen because it had the lowest median error rate, even though 24-day day simple mean had a very slightly lower mean error rate (2.5612 vs 2.5654).

  • By "almost all polls," I exclude Zogby Internet polls (because they are so inaccurate), Strategic Vision and Columbus Dispatch polls (because they are conducted by mail).  Other than that, I include partisan polls, internal campaign polls, and multiple polls from a single polling firm.  Tracking polls are separated into individual polls depending on the number of days the poll was in the field.  For example, a three-day tracking poll will be broken into multiple three-day polls, a four-day tracking poll into multiple four-day polls, etc.

  • By "majority of their interviews conducted during the final 25 days of the campaign," I mean every poll where at least 50% the days the poll was in the field fell during the final 25 days of a campaign.  This resulted in the following cut-off dates for the 50% threshold:

    • 2004: October 8th
    • 2005: October 15th
    • 2006: October 14th
    • 2008: October 10th
    • 2009: October 9th
    • 2010: December 25th, 2009 (Massachusetts special election)

  • At least two polls are used for every campaign, even if less than two polls were conducted during the final 25-days of a campaign.

  • Finally, I only looked at campaigns from 2004-2010 where the final predicted polling margin was less than 18.50.  This is because I could not find a comprehensive list of polls from 2002 or earlier, and because I am simply not interested in forecasting massive electoral blowouts.

  • For 2004 and 2005, the polls used in the calculations were taken from Real Clear Politics.  For 2006-2010, the polls used in the calculations were taken from Pollster.com

Tags: , , , , (All Tags)
Print Friendly View Send As Email

When the polls you're averaging are garbage (0.00 / 0)
the result's still garbage.

I'll point to Wisconsin, where Magellan categorically excluded 18 and 19 year olds (they used 2008 voter lists) and Badger had only 17 of their 500 sample under 30.



This is a Test of the Emergency Free Speech System. This is only a Test. In an actual Free Speech Emergency, I'll be locked up.


Well sure (0.00 / 0)
The polls are the only data points. And, if those are flawed, the projection will be, too. That goes without saying.

However, a median error of only 1.75 across 144 campaigns is pretty accurate.


[ Parent ]
Excellent.... (0.00 / 0)
...but you misspelled "safe" as "sage" in the blurb below the without Rasmussen chart.

I'd like to know (0.00 / 0)
how the polling window (10-day, 25-day, etc.) accuracy changes depending on the early voting rules of the election.  For instance, if it's an all-mail ballot with a 25-day window, that's an obvious reason your 25-day unweighted mean would be more accurate than the weighted averages (in fact, that's so obvious I'm surprised the guys at 538 haven't thought of it).  If early voting is open to all, one would expect to see a slight lean toward more recent polls (since more people will vote on election day than will vote early).  If early voting is restricted to true absentees, one would expect a somewhat greater lean toward the more recent polls, for the same reason.

Do you have any data on this?  If not, how hard would it be to get?  It's quite possible it doesn't matter, but it seems like it would be at least worth investigating.

The Crolian Progressive: as great an adventure as ever I heard of...


No data (0.00 / 0)
I never looked into that. But, I know Oregon is mail-only, and this methodology has been hyper-accurate there. Across the four Oregon campaigns measured here, the median error was 0.31, and the mean was 0.42.

That is only four cases, so that it for what it is worth.


[ Parent ]
Cool (0.00 / 0)
What's the accuracy gap for Oregon between your methodology and that of the other polling gurus?  Is it greater than the average state gap?  That would be interesting information to have.

Another test case would be Washington State, where 88% of the population (in 2006) votes by mail.  

The Crolian Progressive: as great an adventure as ever I heard of...


[ Parent ]
One more request . . . (0.00 / 0)
Could you possibly fix the Kentucky rows?  It's really hard for me to wrap my mind around the concept of Rand Paul running as a Democrat.  Thanks!

Whoa (0.00 / 0)
Oh yeah, will fix.

[ Parent ]
there's a better source for forecasts in the world (0.00 / 0)
Here:

http://www.garyspivey.com/

Plus, he has better hair.


But does it work at this point in the season? (0.00 / 0)
I appreciate all the work that you have done and that you are trying to get an accurate sense of how each campaign is going rather than one biased by rose-colored glasses or pessimism. However, I don't think you've really thought through your methodology.

The methodology is extremely simple: just take the simple mean of almost all polls that had the majority of their interviews conducted during the final 25 days of a campaign

Your methodology seems to work great for predicting the election on election day. But we're still 3 months away from election day. Does it work as well 3 months before an election? If not, then why are you making predictions now? It seems that your methodology is only as good as you say on election day. Before that, it might be a terrible methodology.

The criticism of Rasmussen is that they skew their polls in advance of an election to bias opinion in favor of Republicans, then they poll more accurately the last few days to demonstrate that they actually can do accurate polls. Your methodology does not address this potential hazard at all.

There may be other natural or devious biases in polling done months before an election that your methodology does not catch.


The point is to get something (0.00 / 0)
Relatively accurate, but easy to compute.  It's not to get the most accurate; that would involve spending a ton of money to conduct one's own polls.  You could call it lazy, but you could also call it a useful heuristic.

The idea is that biased polling firms will eventually be caught out, which is why Strategic Vision and Research 2000 are removed.  There is also probably a built-in assumption that there are a sufficient number of competing polling firms that firms have an incentive to be honest, but obviously market forces don't always work like the economics textbooks say they do.

So far, it works sufficiently well for Chris Bowers to not find it useless.  Admittedly, it might skeeve some people because this is exactly how various financial instruments were created, finding correlations that seem to have predictive value without developing a theoretical explanation for why it is so.  In the absence of a clearly better alternative, I don't see a problem with Chris Bowers sticking with his methodology until it fails.

Things You Don't Talk About in Polite Company: Religion, Politics, the Occasional Intersection of Both


[ Parent ]
How do you know it's relatively accurate? (0.00 / 0)
There's no way to test it, as the election is not being held now. Chris has only looked at data from the last 25 days before an election. What does that tell us about data from the last 25 days at any point during the cycle? I just don't know.

But I would say this: One should expect polls from 3 months out to be less accurate than polls from the last 3 weeks. And given that, one could propose that the win% are likely not as clearcut as they seem.

And by the way, this could be checked, with a bit of effort. Just run the same analysis Chris ran on the polls 25 days out, but on polls in July. I'd be interested to see how accurate they were.

DemConWatch


[ Parent ]
I don't see why it would be less accurate now (0.00 / 0)
Unless polling is less accurate now than in October, the methodology shouldn't be any less accurate now than in October.

All I can do is tell you where things stand right now. As far as predicting how the campaign will change, I can't do that. It would be pure guesswork.


[ Parent ]
Accurate enough? (0.00 / 0)
I understand the desire to have a simple, but fairly accurate predictive tool and I greatly appreciate Chris' efforts to develop one. But I fear that his methodology will lead to pessimistic predictions months before an election that will then discourage activists from working on some winnable campaigns. Without their efforts, these winnable campaigns may be lost -- especially since this methodology is so vulnerable to Rasmussen's avalanche of polling and Rasmussen seems biased against Democrats.

Since progressives usually require a strong activist effort to win, a pessimistic prediction months before an election that discourages activists is likely to hurt progressive candidates more than others.


Should progressive energies be directed only towards those campaigns deemed "winnable"? (4.00 / 1)
I personally think that liberals should only work for people they actually want to see in the Senate, irrespective of how close or far they are to winning.  Screw the centrists, and the DSCC rhetoric about needing to save whoever.

Find a campaign you're enthusiastic about, pour your heart into it and let the chips fall where they may.  There's no point in working hard in a close campaign to elect a moderate who'll just backstab you in the end, no matter how much more "winnable" that campaign might appear to be.


[ Parent ]
A few more typos, so you know (0.00 / 0)
Elaine Marshall is misspelled as "Marhsall" in both charts.

The Floridian Democrat is Jeff Greene, not Green.


Arithmetic (0.00 / 0)
The example you give below is not symmetric:


Dem Candidate A trails by 2.25%. Given that 58 of the 144 campaigns have had error rates greater than 2.25%, and that the error has an even chance of breaking in favor of Democrats or Republicans, divide 29 by 144 to arrive at a 20% chance of Democratic victory in that campaign, if election were held today.

Dem Candidate B leads by 1.33%. Given that 91 of the 144 campaigns have had error rates greater than 1.33%, and that the error has an even chance of breaking in favor of Democrats or Republicans, divide 45.5 by 144, and add 50% (because the Dem is ahead), to arrive at a 68% of Democratic victory in that campaign, if the election were held today.


In the first case, you assume the GOP candidate has a 100% chance of winning, but you give a 20% chance of the opposite result, so the GOP candidate is given an 80% chance of winning.

In the second paragraph it says take 45.5/144 + 50%, which would give about 72%, which is incorrect. Like the first case, it should be 100% - 45.5/144 = 68%, which is what you computed.

Your arithmetic is correct, but the textual explanation is not.

By the way, I certainly appreciate the transparency of your method. However, what do you mean by "almost all polls?" The notes do not clarify the "almost."

ec=-8.50 soc=-8.41   (3,967 Watts)


thanks for the clarification (0.00 / 0)
Thanks for the clarity on the formula.

But, I did explain "almost  all polls" in the notes. The polls not included are Zogby Interactive, Strategic Vision, Research 2000 and Columbus Dispatch.


[ Parent ]
Polling and Odds and Palm Reading (0.00 / 0)
A poll based on behaviour is what I would like to see. Who has and who has not distorted information into sound bites to further confuse logic.  Concluding in a poll of who deserves to represent the American people.
I have concluded that if this Nations voters produce a even more split government, the jig is up, this country is done. The way I see it this President is a Democrat and he is not up for election so in November he needs to win full control of the House and Senate. This Country can not go sideways any longer.
As a one time Republican, I will not be voting for any GOP candidate not even my brother-in-law. The party of NO is a disgrace and to act as if any State's people are justified in imposing on the rest of this Nation one of their bullshit artists is unacceptable.
How to get a little since into these regressive States would be time well spent. Never give up no matter what the polls indicate.      

What happens in a tie? (0.00 / 0)
When two candidates are exactly even, as is the case with Sestak vs. Toomey in the non-Rasmussen chart, which side do you award that seat to?

Also, do you think it's worth it to include, either in the charts themselves or in a blurb preceding them, info on the four non-competitive races that are likely to switch partisan control - ND, DE, IN and AR?  That'll help for keeping track of how many seats we'll be at.  A lot of the times I count how many Democratic seats will be lost based on the chart, and then get confused because I forget about the 4 seats we're losing right off the bat.


A tie counts as one-half a seat (0.00 / 0)
A tie works the same as all other seats.  The percentage chance of winning the seat equals the same percentage of a seat gained.  So, if Sestak has a 50% chance of winning the seat, that counts as 50% of a Senate seat.  If Murray has a 93% chance of winning, that counts as .93 of a Senate seat. If you add up the win percentage chances in the last column, you will arrive at the total number of seats the Democrats are most likely to win in those races.  

[ Parent ]
USER MENU

Open Left Campaigns

SEARCH

   

Advanced Search

QUICK HITS
STATE BLOGS
Powered by: SoapBlox