Who Turned Out to Vote in 2022?

Did you vote in the 2022 U.S. midterms? Before the election ever happened, Civis Analytics built models to predict whether you were a sure bet to cast your ballot, or likely to sit out the election instead. 

In fact, over the course of the last year, Civis built models predicting not only which registered voters were likely to vote, but also the general characteristics of the substantial number of folks likely to register late in the cycle. These models were used by our clients to predict the outcomes of races and to allocate resources for persuading likely voters to support their candidate.

Now that the dust has settled and the ballots have been counted, we can start to answer questions like:

The two primary applications of a turnout model — reaching out to potential voters and predicting outcomes — require slightly different choices for optimal modeling. For that reason, we developed both an outreach model and a projection model.

For the outreach model, we’re interested in ordering registrants from most to least likely to vote. This way, campaigns can cut lists that either identify likely voters for persuasion campaigning, or identify unlikely voters for get-out-the-vote outreach. For this model, we include variables that might result in bias by party, such as primary vote history. Whether or not someone voted in the 2022 primary is a very strong predictor of whether they will also vote in the general election; however, in cases where there was a Democratic primary for a Senate seat but no Republican primary, Democrats were rated more likely than Republicans to vote. This is okay for the purposes of outreach models, since our clients are largely contacting Democrats, but we do not include these variables in projection models to avoid partisan bias.

For the projection model, we use a similar but more limited set of variables, and include a model for who is likely to register later in the election cycle, up to Election Day. This model for new registrants computes weights to apply to current registrants when aggregating to predict the total number of registrants on Election Day. For example, a 20-year-old registered to vote as of Oct. 1, 2022 might receive a weight of 1.3, indicating we believe there will be 30 percent more 20-year-olds registered on Nov. 8; on the other hand, an 80-year-old might receive a weight closer to 1, since late registrants are disproportionately young. We trained the models on 2018 registration data from several registration snapshots throughout that year, and used a small number of variables, like age, as model inputs.

We independently modeled eight battleground states: Arizona (AZ), Georgia (GA), Michigan (MI), North Carolina (NC), New Hampshire (NH), Nevada (NV), Pennsylvania (PA), and Wisconsin (WI). Each had closely-watched Senate and/or gubernatorial races, so we spent extra time fine-tuning the predictive variables for these models, splitting the remaining states into groups based on common variables contained in the state voter files and on laws allowing for automatic or same-day registration. For the analysis in this post, we use data from all battleground states except NH, which as of this writing has not released individual-level vote history data.

The job of a turnout modeler is made easier by having nearly complete information about each individual’s vote history. However, it’s also made more challenging than many other types of models used in politics because the training data is from a different election cycle. For 2022, we train the model on 2018 data; however, simply applying the same 2018 formula to 2022 ignores their different political environments. A Republican was president during the 2018 midterms, while a Democrat was president during the 2022 midterms, and historically speaking, this has a large effect on results. Without any additional information, the simple model would almost certainly be rosier for Democrats than it should be, due to the strong Democratic conditions of the training data.

To account for this difference in political headwinds, we implemented a logit shift, a calibration method wherein the average score within a given subgroup is shifted by an amount determined by an external benchmark dataset. Essentially, we used our model to predict turnout in the NJ and VA gubernatorial races in 2021 using the 2017 election as training data, analyzed how much the model missed, and then shifted our 2022 midterm predictions by that miss. This approach assumes that the same political climates persisted from 2017 to 2018 and from 2021 to 2022, and that NJ and VA electorate shifts are representative of national shifts. Testing the same method on previous elections supported this as a reasonable assumption. 

However, as 2022 went on, signals from polling and special elections indicated that the Dobbs decision — the U.S. Supreme Court’s June 2022 ruling to overturn nationwide abortion protections — might have caused a major shift in the electorate, making the voting population notably different from 2021. We analyzed multiple sources for any measurable post-Dobbs shift in the likely electorate, and determined that our 2021-based shift was too large. The original shift moved the likely electorate to being more Republican and white, and the correction we computed for the impact of the Dobbs decision lessened the impact of that shift.

We have nearly complete individual vote histories for seven battleground states. Those states can tell us if the people our model said were more likely to vote really did vote, as well as whether, on aggregate, we predicted the right composition of the electorate across demographics like race, education, and age. We can also compare those aggregates to 2018 to see how the electorate shifted since the previous midterm. Throughout this analysis, we show results using model scores from October 2022, the last time we produced scores before the election.

The metric of interest for the outreach model is ROC-AUC, a measure of how well true positives are associated with high model scores (and true negatives with low model scores). Think of the turnout model in terms of how accurately it orders people in their likelihood to vote. Are people the model scored highly more likely to have voted than low-scored registrants? A perfect model should have a ROC-AUC of 1, where all of the high-score individuals voted and all of the low-score individuals didn’t. A model making random guesses should have a ROC-AUC of about 0.5. 

The chart below shows the statewide ROC-AUC for each battleground state, along with the same metric for the turnout model produced by a competing modeling firm:

These are similar to the values we achieved when testing the models on past elections, so one immediate takeaway is that nothing too surprising happened in 2022.

More interesting is seeing how well we ordered folks within different subgroups, and to compare that to the competitor’s model. The plots below show ROC-AUC broken down by age group: While our model is weakest among young people, that’s also where we differentiate ourselves most sharply from the competition. We can also see from this plot that the Dobbs correction was helpful for this metric:

In Georgia and North Carolina, where we have self-reported race information, we see our models perform best when predicting the turnout probability for larger racial and ethnic groups, but not as well for smaller minorities, which indicates bias favoring the larger classes in lists created from the turnout scores. However, no class is much of an outlier, and in all cases, we outperform the competitor model, with small improvements gained from the Dobbs shift:

Our models also perform better than our competitor for both college-educated and non-college-educated individuals. Further, the models don’t perform significantly differently between these classes, so lists cut from these scores won’t be biased heavily in favor of more educated voters, for example. Note that for the plot below, individual educational attainment is based on Civis’s education model:

The goal of the projection model is accurately predicting the relative turnout rates — and therefore the relative electorate composition — of various subgroups. This analysis also lends itself well to comparisons across elections. For instance, it allows us to compare the 2022 electorate to that of 2018, the year when our models were trained. How did the composition of the electorate change this year, and were our models able to capture that change? The plots below explore these questions in the seven states of interest. 

A general trend in American politics is that rural voters tend to turn out at higher rates than people living in cities. In all of the states we analyzed, this rural/urban turnout gap was larger in 2022 than in 2018. Turnout rates were generally lower than 2018 in urban and suburban areas, and closer to 2018 levels in exurban and rural areas. (The exception is Arizona, where overall turnout was within 1 point of 2018, with slightly higher turnout in suburban and exurban areas, and slightly lower turnout in urban and rural areas.) 

The plot below shows the turnout rate relative to the state average for each classification of population density. For example, 55 percent of registered Arizonans voted in 2022, but only 43% percent of urban-dwelling Arizonans voted, so that group is shown as 0.43 – 0.55 = -0.12 in this plot. For each state, the navy 2022 relative urban turnout rate is further below average than the light blue 2018 rate, and the opposite is true for rural voters. Our models picked up on that trend, but didn’t quite capture the full shift toward rural and exurban areas as compared to 2018:

One key statistic is the fraction of likely voters who voted in the last midterm (or the last presidential election) expected to vote in an upcoming election. This can be used as an additional weighting benchmark for polling, as poll respondents are more accurate in recalling whether they voted in a recent election than predicting if they will vote in the next one. Our model did an excellent job at getting this breakdown correct, as shown in the charts below.

Notably, this breakdown differs markedly from 2018. The reason is clear: The 2020 presidential and 2018 midterm elections posted abnormally high turnout rates, with 71 percent and 57 percent of registered voters turning out nationally, respectively. Four years earlier, the 2016 presidential and 2014 midterm elections had turnout rates of 63 percent and 39 percent, so a lot more infrequent or new voters turned out in 2018. In 2022, the pool of infrequent and new voters was smaller. This difference between the training and scoring year data makes predictions harder, but for most states, our model picked up on this difference:

Nevada is a clear outlier in these charts, which is due at least in part to the disproportionately large share of residents who have registered to vote in the state within the last two years. For those who moved from out of state, imperfect record matching means vote histories from previous residences might not be attached to their NV file. Better matching of records for voters who move to NV would be a huge benefit to predicting turnout in that state.

Going into Election Day, our model predicted turnout levels for voters under 40 and over 70 similar to 2018, with slightly suppressed turnout among middle-aged Americans. For young Americans, this was incorrect for each state we looked at, except for Michigan, where voters under 40 made up about the same share of the electorate as in 2018. In the other five states (excluding Wisconsin, where age is not listed on the voter file), young people dropped from their 2018 turnout rates. Conversely, 70+-year-olds made up an even greater share of the electorate than our model predicted. This can be seen in the plot below, where the light navy 2022 distribution is skewed more toward older voters than the dark navy 2018 distribution: 

In all four states, the electorate included a lower share of college-educated voters than in 2018, based on Civis’s education model to estimate educational attainment for each voter. Our model mostly captured this change from 2018, but slightly under-predicted the share of college educated voters in each state, similar to our competitor’s performance:

For our final model predictions, we shifted turnout rates a little lower for Republicans and white Americans, based on an analysis of polls and special elections after the Dobbs decision. In Georgia, the Dobbs correction had a clear benefit in predicting the racial composition of the electorate. Without that correction, our models were over-predicting the proportion of the electorate that is white and under-predicting the fraction that is Black. With the correction, however, the racial breakdowns are very close, and we fared better than our competitor. In the rest of the states, the Dobbs shift is smaller, but generally moves the racial breakdown in the wrong direction:

In states where we have party registration, the Dobbs effect was helpful for getting the Democratic share correct, but led to slightly under-predicting Republican turnout and over-predicting the share of unaffiliated and third-party voters. However, our model still accurately predicted the relative composition of the electorate:

This can also be seen if we look at predicted vs. actual turnout rates, plotted as a calibration curve by party, shown below for states with party registration. In the plots below, we bin the predicted turnout rate on the x-axis, and plot on the y-axis the actual turnout rate for individuals scored in that bin. For all plots, we see an over-prediction of the statewide turnout rate: We thought more voters would cast their ballots this year, and so did our competitor. But for predicting the composition of the electorate, we want to miss by the same amount across parties. 

As seen in the first column, in each state our competitor missed more for Democrats than for Republicans. If the Civis model hadn’t used the logit shift recalibration using 2021 data, we would have also over-predicted Democratic turnout relative to Republican, as shown in the second column. In the third column, we additionally see that if we had incorporated the 2021-based shift without also applying a correction for the Dobbs decision, we would have a model biased slightly against Democrats. The final column shows the model after both the 2021-based logit shift and the Dobbs correction, demonstrating much stronger alignment across Democrats and Republicans. (We continue to investigate the reasons for our overestimation of unaffiliated voter turnout relative to those registered with a major party.)

As discussed in the previous section, our models performed below expectations in the top-line turnout level — i.e., the fraction of registrants in a state who voted. Our models overshot this number by up to 10 points in some states. 

This wasn’t a target metric for us, so we didn’t optimize for it during model development, since most applications of the model depend simply on the ordering and composition of likely voters. However, we have done some experimenting since the election, and found that building a “de-meaned” model that takes into account an individual’s vote history, as well as the overall turnout rate during the elections for which they were registered, greatly improves our top-line agreement with the actual turnout in each state. 

The plots below depict actual and predicted turnout rates plotted for each county. The size of each point represents the number of registrants in that county. In a perfect model, all of the data points would be centered on the dotted line. As you can see, de-meaning our data improved the top-line turnout rate predictions considerably:

The reason this strategy works is because elections with different turnout rates have different implications on whether the people who voted are likely to vote again. If an individual voted in an election with low turnout, it might mean that person is more highly motivated to vote than their fellow citizens, and their predicted probability of voting in future elections should be increased. However, if someone voted in an election where almost everyone voted, that data point is not as predictive of what that person might do in an election with lower turnout.  

2018 was a historically high turnout year. Using that as training data, we projected that 2018 midterm voters would be extremely likely to vote in 2022. However, because such a large number of voters turned out in 2018, this predictor wasn’t as strong in a more typical turnout year like 2022. By subtracting the average turnout (“de-meaning”) for each election year, we put each election on a more even scale, so extrapolating to other points in time isn’t so perilous.

Our models performed well at the individual level and in the aggregate — typically, better in both categories than our competitor. For the next election, we plan to build turnout models in a similar way to this most recent election, while adding in new vote history variables accounting for the average turnout in each election. We also continue to refine and create new demographic models, which are building blocks for predicting turnout and give Civis a competitive advantage.

Looking ahead to 2024, turnout will undoubtedly be higher, as it is for every presidential election, and the composition of the electorate will differ from 2022. Nonetheless, this most recent midterm provides some lessons, namely:

Turnout models such as those built at Civis offer campaigns and prognosticators the fine-grained data they need to reach potential voters and predict the outcomes of races. This level of precision and detail empowers users to optimize their spending and impact key electoral outcomes. The results shown here indicate that the methodology we employed produced robust predictions. With iteration using lessons learned from this analysis, we will further improve these models moving forward into 2024 and beyond.