Adventures in Tracking Polls

by: fladem

Mon Sep 08, 2008 at 11:35

I don't like cherry picking polls.  I think that is prescription for re-enforcing your own prejudices.  But this morning's USA Today/Gallup Poll demands comment.   It is, to borrow a phrase from the New York Times, yet another lesson in the "perils of polling".

The first time serious questions were raised about Gallup's tracking polls was 1996.  Consider the following quote from a New York Times account in October of 1996 highlighting the absurd volatility Gallup was finding:

''The latest CNN/USA Today Gallup tracking poll shows the race nationwide is tightening,'' Wolf Blitzer, a CNN correspondent, said on the program ''Inside Politics'' on Saturday. ''This single-digit lead is half of what it was earlier this month.''

Appearing on the program, Scott Reed, the Dole campaign manager, said: ''That's consistent with our internal numbers. And what you've seen here in the last 10 days, the race has closed.''

But by Tuesday, Judy Woodruff, also on ''Inside Politics,'' declared that Mr. Clinton's ''domestic political standing is strong,'' announcing that he had opened his largest lead, 25 points, since the poll started at the beginning of September.

The true height of absurdity was Gallup's tracking polls in 2000.  They were laughable.    Alan Abramowitz of Emory recently highlighted one such example: "On October 24 Gallup had Gore ahead of Bush by one point. Three days later, on October 27, they had Bush ahead by 13".

He went on to note (a great comment on the flip)

fladem :: Adventures in Tracking Polls

No other poll during the 2000 campaign showed anything like the volatility of the Gallup tracking poll and so far Gallup's 2008 tracking poll has shown nothing like the volatility of their 2000 tracking poll. That's probably because they haven't started to apply their likely voter screen to the tracking poll.

A study by Robert Erickson of Columbia concluded:

This article explores how Gallup's likely voter model exaggerates the reported volatility of voter preferences during the campaign. Much of the reported variation in candidate preference reported by Gallup in that election is not due to actual voter shifts in preference but rather to changes in the composition of Gallup's likely voter pool. The findings highlight dangers of relying on samples of likely voters when polling well before Election Day.

Over the last year I have studied polling history a fair amount.  One conclusion I have reached is that medians are far more useful than averages.   In any cycle there are outliers, and the inclusion of those polls in averages will inevitably skew the results.  I can point to other examples of this (Pew showed Bush with 16 point lead after the 2004 GOP convention when no one else found a lead larger than 11).

There can be no question that McCain has received a substantial bounce - and it's cause for concern.  But  the biggest concern I have right now is that two pollsters with questionable records are driving the narrative.   Caution is appropriate in over reading polls right now.  

Tags: (All Tags)
Print Friendly View Send As Email

I want to know more about the LV screens (4.00 / 2)
each pollster is using. We have record numbers of new Democratic registrations in many of the battleground states. I assume that most of the time, someone who has never voted before would fail a LV screen.

However, I suspect Obama's turnout machine will be more successful than past Democratic efforts.

Join the Iowa progressive community at Bleeding Heartland.

We are regestering 500 (4.00 / 5)
new Democrats a week in Hillsborough Country alone.  It will be next to impossible to include these new voters in polls.  

[ Parent ]
Iowa Democrats had a 90,000 advantage (4.00 / 3)
in registrations as of June 2008. Probably that advantage has increased over the summer, as the Obama campaign has canvassed in well over 100 cities and towns.

Registered Republicans slightly outnumbered Iowa Democrats when Gore carried the state by about 5,000 votes (out of 1.3 million cast) and Bush carried the state by about 10,000 votes (out of 1.5 million cast).

Join the Iowa progressive community at Bleeding Heartland.

[ Parent ]
Bush won florida by 300,000 votes (4.00 / 1)
Last I saw we only had a 100000+ increase in Dem registration over R.

Over at 538, Nate Silver has an interesting hypothesis which I think may pan explain this time why 1. the Gallup LV voter screen may actually catching something real and 2. maybe the huge generic Dem advantage has been an artefact of the shame of right wing voters who refused to admit they are for Bush in this era of epic Republican failure. Sarah Palin has both energized them and made them proud again.  They are picking up the pollster's calls now. They were avoiding them before.

I think we have a fight on our hands.  They are going to activate their turnout machine, now they have enthusiastic recruits. It's not only going to affect the presidiential but downticket races as well.

"Incrementalism isn't a different path to the same place, it could be a different path to a different place"

[ Parent ]
That is an interesting (0.00 / 0)

In 2004 the Bush signs in Florida appeared after their convention.  The bloomed like flowers after a summer rain.  

[ Parent ]
I know I was there (0.00 / 0)
My daughter ran Palm Beach county for the Kerry felt different after the convention than before.

Anybody in a pickup truck is an's true.   Palm Beach had huge new numbers of both Dems and Repubs....hunfreds of thousands newbies.  Most of the R's were construction workers involved in the Fla building boom. They were from the deep south originally...Maybe some of them went honme.

"Incrementalism isn't a different path to the same place, it could be a different path to a different place"

[ Parent ]
Thank you fladem (4.00 / 3)
As one who has been at a loss for words regarding recent polling, this is a little reassuring.

Gallup's LV adventures in 2000 (4.00 / 2)
Abramowitz advises against using LV screens too early in polls, but those crazy double digit Gallup swings in 2000 were still happening in late October, which was only two weeks before the election!  So their LV screen at that time was probably worse than useless, since it was still producing huge shifts up until the final days of the election.  

One reason why I am suspicious (4.00 / 4)
of both Rasmussen and Gallup is the low level of undecided.
If you look at the non-Rasmussen State polling, you find about 9% undecided - higher than in both Gallup and Rasmussen.  

[ Parent ]
That seems about right (4.00 / 1)
I'd say about 8-10%

[ Parent ]
Low undecideds (4.00 / 1)
and high swings do not add up.

John McCain opposes the GI Bill.

[ Parent ]
Dead right (0.00 / 0)
which is why to be carefull reading them.

[ Parent ]
Polling is polling (0.00 / 0)
Its much more of an art than a science.  You're trying to predict the behavior of a couple hundred million Americans (will they vote or not?) and then ascertain the preferences of a random sample.  Its fraught with peril by its nature and people should take the data with a shaker of salt, whether its good news or bad.

That said, I don't think there is any reason to single out the tracking polls.  They probably have more reliable data than any of the outfits that poll nationally once a month and certainly better than state polls, which are notoriously bad.      There are definitely pitfalls to likely voter screens, but there are problems with registered voter polls too.  

The long and the short of it is that we're in for a dogfight and its probably a tossup at this point.  Trying to say anything more precise about the state of the race is just shouting into a wind of statistical noise.

John McCain: Health insurance for low income children represents an "unfunded liability."

Fear not the Daily Tracking Polls ..... (4.00 / 1)
I've just written a longer post on Nate Silver ( about this.

Basically, any daily measurement needs to be weighed with its underlying trend to get an accurate mean measurement. That is independent of whether the daily measurement is honest or not (and of course, it may not be, so you have two reasons to distrust it).

The daily measurements are like maritime surface phenomena, like waves, wind etc. But there is also an underlying current - you have to allow for both to steer properly.

At the moment we just do not know the underlying trend ("the fundamentals of the election") because McCain's convention bounce is the equivalent of a hurricane. We need that to blow out (a week, maybe less) before we can take stock again.

So, let there be no panic because the tracking polls are not in our favour! Obama's numbers are just under his former minimum (45.3%) on the RealClearPolitics average of all the current polls, while McCain is at his all-time maximum (47.3%). It's a bubble, I tells ya!

RealClearPolitics and are the best sites to keep track of the trends, but at the moment, the McCain convention hurricane is skewing the measurements. still have Obama in a 3% lead, presumably weighs the Polls differently than RealClearPolitics. Can't say I really understand that figure myself, but they seem to have the same polls as RealClearPolitics.

Incidentally, the 54% for McCain on the USAToday/Gallup poll is the highest polling figure either candidate has got so far. Go see it on - it is a clear outlier.

I am sure the Obama team have their own polling data and focus groups. If the game has indeed changed, his team should know first. Then he needs to re-focus and re-ignite his campaign.

I sense a decline in morale among the Obama folks. But I also think people are reading too much into the daily polls - don't trust those polls. Even if they are 100% honest, they can still skew the measurement.

RCP versus (4.00 / 1)
I believe pollster uses a weighted regression while RCP I think just uses the simple mean.'s method I think has more 'memory' built into their estimate.  

[ Parent ]
At this point (0.00 / 0)
one poll is skewing the RCP numbers.

I use both sites: but have problems with each.

My problem with is they tend to include older data that events have rendered obsolete.  They also include the online Zogby polling: which is junk.

My problem with RCP is their preference for Likely Voters in their national polling numbers.  

[ Parent ]
Re: RCP & Pollster (4.00 / 1)
It depends on the window you use to calculate trend.

RCP use a short window, so its metric is quite noisy and responds quickly (but sometimes wrongly) to short-term changes. uses a longer window so it  has a high inertia factor and changes very slowly.

As a professor of mine used to say:

"You pays your money and you makes your choice!"

[ Parent ]
did you see? (0.00 / 0)
The Great Orange Satan just announced that they're starting a daily tracking poll from R2K along with a bunch of other polls.  Should be cool, especially since kos is making the entire poll (day by day crosstabs, etc) public.

I'm glad it's done


Open Left Campaigns



Advanced Search

Powered by: SoapBlox