Existing research suggests that news events can increase the amount of content online about that topic. However, there is a gap in current research about how much big news events impact the volume of online misinformation. Our goal was to help fill this gap by studying the association between the May 2022 Supreme Court leak, the final June 2022 decision, and the volume of abortion misinformation online around those events. We used Twitter data to do this analysis.

Our results found that the May leak and June decision were both associated with an increase in misinformation online. 

Using Google Trends and CrowdTangle data, we first confirmed that the number of posts about abortion increased following the Supreme Court leak and decision on Twitter, Facebook, and Instagram. With that overall content increase in mind, we wanted to understand if there was a change in the volume of misinformation. 

Misinformation increases overall

To assess misinformation change, we queried Twitter’s historical search API and downloaded all tweets containing the word “abortion” (excluding retweets) from April 1 - July 31, 2022. We sampled 25 tweets per day for each of the four days before and after both the Supreme Court leak and decision. We then qualitatively analyzed the prevalence of tweets that contained misinformation, using two coders

We found that misinformation jumped from 4.7% of the tweets surveyed before the May 2, 2022 leak, to 8.8% of the tweets surveyed in the four days afterward. This suggests a near doubling of misinformation in content sampled from Twitter following the leaked decision. While we observed an increase in misinformation following the leak, this increase wasn’t big enough to be statistically significant. This may be in part due to our small sample size of just 100 tweets across each 4-day period and underscores the need for further research. 

The same analysis of content surrounding the Supreme Court decision found that misinformation increased by about 35% – from 6% to 8% – during the four days surveyed before and after the June 2022 Supreme Court decision. 

Misinformation spikes were also higher after the events

As the figure above shows, the volume of misinformation reaches a sharp peak that then rapidly falls after the leak and decision. An outlier test we ran on these peaks yielded statistically significant findings. 

We found that the highest point of misinformation after the leak and decision were statistical outliers. This means that that data point is unlikely to have come from the previous distribution, and that the spike is very likely to have been driven by the event itself. 

Specific misinformation terms showed similar results 

We ran these same analyses on three search terms about abortion to further examine  how the volume of misinformation changed within specific subtopics about abortion. These three search terms were:

1. “chemical abortion”
2. “herbal abortion”
3. “abortion” AND “pill revers”

We selected these specific search terms due to their known association with misinformation based on subject matter knowledge expertise, and their increase in volume across Facebook, Instagram, and Twitter based on Crowdtangle data analyzed in the days following the Supreme Court leak and decision. We then qualitatively coded all Tweets made with each phrase in the four days before and after the leak and decision as either “misinformation” or “not misinformation” and ran a t-test on the difference between the two samples in terms of misinformation. 

Our results show “herbal abortion” and “abortion” AND “pill revers” to be associated with higher volumes of misinformation after both the leak and decision compared to before the Supreme Court leak and decision. 

The term “chemical abortion” was not associated with higher volumes of misinformation before and after the leak and decision. This may be due to the fact that it had the greatest amount of noise in the data, and had a higher than expected volume of misinformation sustained for a few days after the decision. 

None of these findings were statistically significant, which, similar to our first analysis, may be in part due to the small amount of data sampled per day, and the noisiness of the data. As a result, we can’t say based on this test that there’s a clear sustained increase from before the Supreme Court leak or decision to after those two events, respectively. 

Like with the “abortion” term data, we also ran outlier detection by analyzing just the highest peak of misinformation associated with each search term compared to the rest of the data. Through this test we found the highest point of misinformation for “herbal abortion” and “abortion” AND “pill revers” to statistically be an outlier. This means that those peak data points are unlikely to have come from the previous distribution, and that the spike is very likely to have been driven by the event itself. 

Conclusion

In summary, we found a non-statistically significant increase in the volume of misinformation after both the Supreme Court leak and decision when examining sampled tweets from the search term “abortion” as well as with two of the three search terms associated with abortion misinformation that we analyzed (“herbal abortion” and “abortion” AND “pill revers”). We also found that the highest peak of misinformation among sampled tweets from the search term “abortion” as well as with two of the three search terms analyzed (“herbal abortion” and “abortion” AND “pill revers”) were statistically significant outliers. 

These findings suggest that there was an increase in overall content about abortion following the Supreme Court leak and decision, and serve as preliminary evidence that there was also more misinformation about abortion within that increased volume. The data also suggest that following large policy decisions and news events, misinformation spikes significantly in the 1-2 days immediately following, and then drops quickly. This finding suggests that platforms could benefit from both 1) planning for large spikes in misinformation by anticipating news events, and 2) reacting quickly to tackle claims in the immediate aftermath of a news event. 

In terms of the substance of the misinformation analyzed, many claims we observed in the four-day period following the Supreme Court leak and decision were the same narratives that existed online before the Supreme Court leak and decision. This finding suggests that there is promise in preparing for potential spikes in mis and disinformation through prebunking and setting contingency plans in place with fact-checks ready for common narratives about different topics (eg. abortion, COVID-19, etc).

‍Looking ahead

More work needs to be done to replicate and validate these findings, as well as to conduct more empirical research on this topic that accounts for the limitations of this study. Ways to account for our limitations and further this research include:

• Using a broader selection of keywords. Doing so would expand the scope of the study and create a dataset with greater representation of sub-topics that fall under abortion. In essence, this would allow for a stronger study that shows how misinformation changes across a range of topics and sub-topics. 
• Increasing the sample size of tweets or posts per day sampled from the broad term “abortion” would allow for more robust data analyses and would strengthen findings by increasing the power. In our study, we sampled 25 tweets per day. Sampling 50 tweets, 100 tweets, or more, per day would strengthen the analysis. .
• Studying a longer time range for qualitative analysis of sampled tweets and specific search terms. In our study, we focused on the 4-7 days following the event. Expanding the time period of analysis would provide a greater understanding of the duration of the effects of the events on misinformation.
• Expanding this methodology to other platforms would shed light on whether these findings are consistent across platforms and how misinformation volumes change based on platform.
• Using additional qualitative reviewers would strengthen the internal validity of the study by increasing both specificity and sensitivity, and working with topic area experts to conduct the qualitative review would provide a more diverse range of expert perspectives that would help to strengthen the analysis.

In addition to improving this study, more work is needed in general to understand how misinformation changes in the wake of other news events so that we can continue to improve our preparation for surges in misinformation – particularly as digital ecosystems risk growing more divisive and as crises grow more likely due to climate change. 

Finally, more research is also needed on the health impacts of this type of misinformation. It’s one thing to track how misinformation is increasing, and another to know what the actual adverse impacts of surges in misinformation are.