Why the 538 forecast has not changed significantly after the debate

Why the 538 forecast has not changed significantly after the debate
Why the 538 forecast has not changed significantly after the debate

According to the 538 average, former President Donald Trump’s lead in national polls has grown by nearly 2 percentage points since June 27, when President Joe Biden delivered a weak performance in the first presidential debate of the election cycle. Biden’s current 2.1-point deficit is the worst position by a Democratic presidential candidate since early 2000, when then-Vice President Al Gore was far behind Texas Governor George W. Bush, according to our retrospective polling average of that race.

PHOTO: An image showing the national average of 538 presidential polls, with former President Donald Trump at 42.1 percent, President Joe Biden at 39.9 percent and Robert F. Kennedy Jr. at 9.5 percent.

The national average of 538 presidential polls puts former President Donald Trump 2.1 percentage points ahead of President Joe Biden.

538 Photo illustration

Our forecast for the 2024 election, however, has changed little. On debate day, we gave Biden a 50-100 chance of winning a majority of the Electoral College votes. On July 5, that chance briefly dropped to 46-100, and now it’s 48-100. At first glance, this lack of change is puzzling: If Biden has fallen from roughly tied to over 2 points in national polls, shouldn’t his chances of winning have dropped by more than a few points?

PHOTO: A screenshot of the 538 presidential projection, which shows former President Donald Trump with a 52 percent chance of winning and President Joe Biden with a 48 percent chance of winning. Robert F. Kennedy Jr. has a less than 1 percent chance of winning.

The 538 prediction for the 2024 presidential election has changed little since the first debate.

538 Photo illustration

Well, not necessarily. The “right” amount of variability in a forecast depends on how much uncertainty you expect between now and the event you’re trying to predict. It’s probably easy for you to predict how hungry you’ll be tomorrow morning, for example – a prediction with pretty low variance (unless you’re on an international trip or running a marathon). But some things, like medium-term stock returns, are much harder to predict.

It’s also difficult to predict what public opinion will look like in four months. In part, that’s because small events — like debates — can cause big changes in the polls. But it’s also difficult because the polls themselves are inaccurate. There are many sources of uncertainty that all need to be combined properly, and forecasters make a lot of informed but imperfect decisions trying to figure out how to do that. (This is true whether you have a mental model or a statistical one, like we do at 538.)

To make things easier to digest, let’s go through our main sources of uncertainty about the election in two parts.

First, there is uncertainty about how accurate the polls will be on Election Day. This is relatively easy to measure: We run a model to calculate how many polling errors there have been in recent years and how strongly these correlated from state to state. This model tells us that if polls overestimate Democrats by 1 point in, say, Wisconsin, they are likely to overestimate Democrats in the average state by about 0.8 points—and by nearly 0.9 points in a state like Michigan, which is demographically and politically similar to America’s Dairyland. On average, we simulate about 3.5 to 4 points of polling bias in each scenario of our election model—meaning that we expect polls to overestimate Democrats about 70 percent of the time either The gap between Democrats and Republicans is 3.5 points at most, and 95 percent of the time we expect the gap to be less than 8 points.

Those are pretty large uncertainty intervals—from what I can tell, they’re about 25 percent larger than those in some other election forecasting models. One reason for that size is that 538’s model more closely follows trends in state-level poll reliability. It’s really that simple: Polls have been worse recently, so we’re simulating more potential bias across states. And while our model could take a longer-term perspective and reduce the bias we simulated, such a model would have performed much worse in 2016 and 2020 than our current version. We think that even if polls end up being accurate this year, we would have preferred to consider a scenario in which poll error is nearly 50 percent larger than in 2020—as was the case in 2020 compared to 2016.

But the second and larger source of election uncertainty is how much poll numbers will change between now and Election Day. By predicting future changes in poll numbers, we effectively “smooth out” bumps in our poll averages when we translate them into Election Day predictions. Let’s think hypothetically for a moment: If a candidate gains 1 point in the poll numbers, but we expect poll numbers to change by 20 points on average between now and November, the candidate’s increased probability of winning will be much smaller than if we only expected a 10-point change.

Today, we simulate an average change in the gap between the two candidates in the average state of about 8 points for the rest of the campaign. We get this number by calculating the state’s 538 average poll numbers for every day of all elections from 1948 to 2020, finding the absolute difference between the poll number on a given day and the average on Election Day, and averaging those differences for each day of the campaign. We find that between 300 days before Election Day and Election Day itself, poll numbers change by about 12 points on average. That means poll numbers change by about 0.35 points per day on average.

It’s true that polls are less volatile than they used to be; from 2000 to 2020, the average contested state saw only an 8-point change in polls 300 days before Election Day, on average. But there are some good reasons to use the larger historical dataset rather than limiting the analysis to the most recent election.

First, it is the most reliable estimate; in election forecasting we work with a very small sample and need as many observations as possible. By looking longer, we can also better account for possible electoral realignments – some of which seem plausible this year. Given what has happened so far in the campaign, it may be safer to leave room for volatility.

On the other hand, simulating smaller polling errors over the course of the campaign gives you a forecast that assumes very little volatility in the polls after Labor Day. That’s because the statistical technique we use to examine errors over time – called the “random walk” technique – spreads opinion changes evenly over the entire campaign. But campaign events that affect voters’ preferences tend to cluster in the fall toward the end of the campaign, and many voters don’t pay attention until then anyway. For that reason, we prefer to use a dataset that prices in more volatility after Labor Day.

As you can see, both the modeled error based on the 1948-2020 elections and the modeled error based on the 2000-2020 elections underestimate the actual poll shifts that occurred in the final months of those elections. However, the underestimation of the variance for the 2000-2020 elections is particularly undesirable because it underestimates the error under study even further than for the most recent elections. Therefore, for our forecasting model, we use the historical dataset, which provides more uncertainty at the beginning, so that we get the right level of uncertainty for the most recent elections later—though we still underestimate the variance in some of the oldest elections in our training set. Essentially, the final modeled error we use splits the difference between historical and recent poll volatility.

Now it’s time to combine all of those uncertainties. The way our model works is to update a previous prediction about the election with the conclusions from the polls. In this case, the model’s starting position is a fundamentals-based forecast that predicts historical state election results using economic and political factors. Then the polls are essentially superimposed on that projection. The model’s cumulative forecast is weighted more heavily toward the prediction we are more certain of: either the one based on historical fundamentals or the prediction of what the polls will say on election day and whether they will be accurate. Right now, we’re not entirely sure what the polls will show in November, and that reduces the weight our final forecast gives to today’s polls.

So the reason our forecast hasn’t changed much since the debate is because a 2-point swing in the race now doesn’t necessarily equate to a 2-point swing on Election Day. The only way to take the polls more seriously is simply to wait and see.

For fun, let’s see what our forecast would say with different model settings. I ran our model four times: once as is (with the 8 points of error remaining), once with a medium poll swing (about 6 points between now and November), once with a fairly small move (4 points), and finally a version with both little poll swing and no historical fundamentals – a pure poll model.

The results of these different model settings tell us that less uncertainty about how the election will play out increases Trump’s chances of winning. That’s because he’s leading in the polls today. So if our prediction of the polls on Election Day is more in line with where they are now, Biden’s lead narrows, away from the fundamentals and closer to what would happen if the election were held today. And if we remove the fundamentals from the model entirely, we get an even higher probability of a Trump win.

And that’s why the 538 forecast was more stable than others: Most other forecasters simply take current polls more seriously. That’s not necessarily the wrong prediction about the future; we just found that fewer errors did not work well in the backtest historical. Many different statistical models can be somewhat helpful in explaining electoral uncertainty, and since there are few historical cases, it is impossible to say that one of us is right or the other is wrong.

Leave a Reply

Your email address will not be published. Required fields are marked *