Abstract
The social media platforms of the twenty-first century have an enormous role in regulating speech in the USA and worldwide1. However, there has been little research on platform-wide interventions on speech2,3. Here we evaluate the effect of the decision by Twitter to suddenly deplatform 70,000 misinformation traffickers in response to the violence at the US Capitol on 6 January 2021 (a series of events commonly known as and referred to here as ‘January 6th’). Using a panel of more than 500,000 active Twitter users4,5 and natural experimental designs6,7, we evaluate the effects of this intervention on the circulation of misinformation on Twitter. We show that the intervention reduced circulation of misinformation by the deplatformed users as well as by those who followed the deplatformed users, though we cannot identify the magnitude of the causal estimates owing to the co-occurrence of the deplatforming intervention with the events surrounding January 6th. We also find that many of the misinformation traffickers who were not deplatformed left Twitter following the intervention. The results inform the historical record surrounding the insurrection, a momentous event in US history, and indicate the capacity of social media platforms to control the circulation of misinformation, and more generally to regulate public discourse.
This is a preview of subscription content, access via your institution
Access options
Access Nature and 54 other Nature Portfolio journals
Get Nature+, our best-value online-access subscription
$29.99 / 30 days
cancel any time
Subscribe to this journal
Receive 51 print issues and online access
$199.00 per year
only $3.90 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
Data availability
Aggregate data used in the analysis are publicly available at the OSF project website (https://doi.org/10.17605/OSF.IO/KU8Z4) to any researcher for purposes of reproducing or extending the analysis. The tweet-level data and specific user demographics cannot be publicly shared owing to privacy concerns arising from matching data to administrative records, data use agreements and platforms’ terms of service. Our replication materials include the code used to produce the aggregate data from the tweet-level data, and the tweet-level data can be accessed after signing a data-use agreement. For access requests, please contact D.M.J.L.
Code availability
All code necessary for reproduction of the results is available at the OSF project site https://doi.org/10.17605/OSF.IO/KU8Z4.
References
Lazer, D. The rise of the social algorithm. Science 348, 1090–1091 (2015).
Jhaver, S., Boylston, C., Yang, D. & Bruckman, A. Evaluating the effectiveness of deplatforming as a moderation strategy on Twitter. Proc. ACM Hum.-Comput. Interact. 5, 381 (2021).
Broniatowski, D. A., Simons, J. R., Gu, J., Jamison, A. M. & Abroms, L. C. The efficacy of Facebook’s vaccine misinformation policies and architecture during the COVID-19 pandemic. Sci. Adv. 9, eadh2132 (2023).
Hughes, A. G. et al. Using administrative records and survey data to construct samples of tweeters and tweets. Public Opin. Q. 85, 323–346 (2021).
Shugars, S. et al. Pandemics, protests, and publics: demographic activity and engagement on Twitter in 2020. J. Quant. Descr. Digit. Media https://doi.org/10.51685/jqd.2021.002 (2021).
Imbens, G. W., & Lemieux, T. Regression discontinuity designs: a guide to practice. J. Econom. 142, 615–635 (2008).
Gerber, A. S. & Green, D. P. Field Experiments: Design, Analysis, and Interpretation (W.W. Norton, 2012).
Grinberg, N., Joseph, K., Friedland, L., Swire-Thompson, B. & Lazer, D. Fake news on Twitter during the 2016 U.S. presidential election. Science 363, 374–378 (2019).
Munger, K. & Phillips, J. Right-wing YouTube: a supply and demand perspective. Int. J. Press Polit. 27, 186–219 (2022).
Guess, et al. How do social media feed algorithms affect attitudes and behavior in an election campaign? Science 381, 398–404 (2023).
Persily, N. in New Technologies of Communication and the First Amendment: The Internet, Social Media and Censorship (ed. Bollinger L. C. & Stone, G. R.) (Oxford Univ. Press, 2022).
Sevanian, A. M. Section 230 of the Communications Decency Act: a ‘good Samaritan’ law without the requirement of acting as a ‘good Samaritan’. UCLA Ent. L. Rev. https://doi.org/10.5070/LR8211027178 (2014).
Lazer, D. M. J. et al. The science of fake news. Science 359, 1094–1096 (2018).
Suzor, N. Digital constitutionalism: using the rule of law to evaluate the legitimacy of governance by platforms. Soc. Media Soc. 4, 2056305118787812 (2018).
Napoli, P. M. Social Media and the Public Interest (Columbia Univ. Press, 2019).
DeNardis, L. & Hackl, A. M. Internet governance by social media platforms. Telecomm. Policy 39, 761–770 (2015).
TwitterSafety. An update following the riots in Washington, DC. Twitter https://blog.x.com/en_us/topics/company/2021/protecting--the-conversation-following-the-riots-in-washington-- (2021).
Twitter. Civic Integrity Policy. Twitter https://help.twitter.com/en/rules-and-policies/election-integrity-policy (2021).
Promoting safety and expression. Facebook https://about.facebook.com/actions/promoting-safety-and-expression/ (2021).
Dwoskin, E. Trump is suspended from Facebook for 2 years and can��t return until ‘risk to public safety is receded’. The Washington Post https://www.washingtonpost.com/technology/2021/06/03/trump-facebook-oversight-board/ (4 June 2021).
Huszár, F. et al. Algorithmic amplification of politics on Twitter. Proc. Natl Acad. Sci. USA 119, e2025334119 (2021).
Guess, A. M., Nyhan, B. & Reifler, J. Exposure to untrustworthy websites in the 2016 US election. Nat. Hum. Behav. 4, 472–480 (2020).
Sunstein, C. R. #Republic: Divided Democracy in the Age of Social Media (Princeton Univ. Press, 2017).
Timberg, C., Dwoskin, E. & Albergotti, R. Inside Facebook, Jan. 6 violence fueled anger, regret over missed warning signs. The Washington Post https://www.washingtonpost.com/technology/2021/10/22/jan-6-capitol-riot-facebook/ (22 October 2021).
Chandrasekharan, E. et al. You can’t stay here: the efficacy of Reddit’s 2015 ban examined through hate speech. Proc. ACM Hum. Comput. Interact. 1, 31 (2017).
Matias, J. N. Preventing harassment and increasing group participation through social norms in 2,190 online science discussions. Proc. Natl Acad. Sci. USA 116, 9785–9789 (2019).
Yildirim, M. M., Nagler, J., Bonneau, R. & Tucker, J. A. Short of suspension: how suspension warnings can reduce hate speech on Twitter. Perspect. Politics 21, 651–663 (2023).
Guess, A. M. et al. Reshares on social media amplify political news but do not detectably affect beliefs or opinions. Science 381, 404–408 (2023).
Nyhan, B. et al. Like-minded sources on Facebook are prevalent but not polarizing. Nature 620, 137–144 (2023).
Dang, S. Elon Musk’s X restructuring curtails disinformation research, spurs legal fears. Reuters https://www.reuters.com/technology/elon-musks-x-restructuring-curtails-disinformation-research-spurs-legal-fears-2023-11-06/ (6 November 2023).
Duffy, C. For misinformation peddlers on social media, it’s three strikes and you’re out. Or five. Maybe more. CNN Business https://edition.cnn.com/2021/09/01/tech/social-media-misinformation-strike-policies/index.html (1 September 2021).
Conger, K. Twitter removes Chinese disinformation campaign. The New York Times https://www.nytimes.com/2020/06/11/technology/twitter-chinese-misinformation.html (11 June 2020).
Timberg, C. & Mahtani, S. Facebook bans Myanmar’s military, citing threat of new violence after Feb. 1 coup. The Washington Post https://www.washingtonpost.com/technology/2021/02/24/facebook-myanmar-coup-genocide/ (24 February 2021).
Barry, D. & Frenkel, S. ‘Be there. Will be wild!’: Trump all but circled the date. The New York Times https://www.nytimes.com/2021/01/06/us/politics/capitol-mob-trump-supporters.html (6 January 2021).
Timberg, C. Twitter ban reveals that tech companies held keys to Trump’s power all along. The Washington Post https://www.washingtonpost.com/technology/2021/01/14/trump-twitter-megaphone/ (14 January 2021).
Dwoskin, E. & Tiku, N. How Twitter, on the front lines of history, finally decided to ban Trump. The Washington Post https://www.washingtonpost.com/technology/2021/01/16/how-twitter-banned-trump/ (16 January 2021).
Harwell, D. New video undercuts claim Twitter censored pro-Trump views before Jan. 6. The Washington Post https://www.washingtonpost.com/technology/2023/06/23/new-twitter-video-jan6/ (23 June 2023).
Romm, T. & Dwoskin, E. Twitter purged more than 70,000 accounts affiliated with QAnon following Capitol riot. The Washington Post https://www.washingtonpost.com/technology/2021/01/11/trump-twitter-ban/ (11 January 2021).
Denham, H. These are the platforms that have banned Trump and his allies. The Washington Post https://www.washingtonpost.com/technology/2021/01/11/trump-banned-social-media/ (13 January 2021).
Graphika Team. DisQualified: network impact of Twitter’s latest QAnon enforcement. Graphika Blog https://graphika.com/posts/disqualified-network-impact-of-twitters-latest-qanon-enforcement/ (2021).
Dwoskin, E. & Timberg, C. Misinformation dropped dramatically the week after Twitter banned Trump and some allies. The Washington Post https://www.washingtonpost.com/technology/2021/01/16/misinformation-trump-twitter/ (16 January 2021).
Harwell, D. & Dawsey, J. Trump is sliding toward online irrelevance. His new blog isn’t helping. The Washington Post https://www.washingtonpost.com/technology/2021/05/21/trump-online-traffic-plunge/ (21 May 2021).
Olteanu, A., Castillo, C., Boy, J. & Varshney, K. The effect of extremist violence on hateful speech online. In Proc. 12th International AAAI Conference on Web and Social Media https://doi.org/10.1609/icwsm.v12i1.15040 (ICWSM, 2018).
Lin, H. et al. High level of correspondence across different news domain quality rating sets. PNAS Nexus 2, gad286 (2023).
Abilov, A., Hua, Y., Matatov, H., Amir, O., & Naaman, M. VoterFraud2020: a multi-modal dataset of election fraud claims on Twitter.” Proc. Int. AAAI Conf. Weblogs Soc. Media 15, 901–912 (2021).
Calonico, S., Cattaneo, M. D. & Titiunik, R. Robust nonparametric confidence intervals for regression-discontinuity designs. Econometrica 82, 2295–2326 (2014).
Jackson, S., Gorman, B. & Nakatsuka, M. QAnon on Twitter: An Overview (Institute for Data, Democracy and Politics, George Washington Univ. 2021).
Shearer, E. & Mitchell, A. News use across social media platforms in 2020. Pew Research Center https://www.pewresearch.org/journalism/2021/01/12/news-use-across-social-media-platforms-in-2020/ (2021).
McGregor, S. C. Social media as public opinion: How journalists use social media to represent public opinion. Journalism 20, 1070–1086 (2019).
Hammond-Errey, M. Elon Musk’s Twitter is becoming a sewer of disinformation. Foreign Policy https://foreignpolicy.com/2023/07/15/elon-musk-twitter-blue-checks-verification-disinformation-propaganda-russia-china-trust-safety/ (15 July 2023).
Joseph, K. et al. (Mis)alignment between stance expressed in social media data and public opinion surveys. Proc. 2021 Conference on Empirical Methods in Natural Language Processing 312–324 (Association for Computational Linguistics, 2021).
Robertson, R. E. et al. Auditing partisan audience bias within Google search. Proc. ACM Hum. Comput. Interact. 2, 148 (2018).
McCrary, J. Manipulation of the running variable in the regression discontinuity design: a density. Test 142, 698–714 (2008).
Roth, J., Sant’Anna, P. H. C., Bilinski, A. & Poe, J. What’s trending in difference-in-differences? A synthesis of the recent econometrics literature. J. Econom. 235, 2218–2244 (2023).
Wing, C., Simon, K. & Bello-Gomez, R. A. Designing difference in difference studies: best practices for public health policy research. Annu. Rev. Public Health 39, 453–469 (2018).
Baker, A. C., Larcker, D. F. & Wang, C. C. Y. How much should we trust staggered difference-in-differences estimates? J. Financ. Econ. 144, 370–395 (2022).
Callaway, B. & Sant’Anna, P. H. C. Difference-in-differences with multiple time periods. J. Econom. 225, 200–230 (2021).
R Core Team. R: A Language and Environment for Statistical Computing, v.4.3.1. https://www.R-project.org/ (2023).
rdrobust: Robust data-driven statistical inference in regression-discontinuity designs. https://cran.r-project.org/package=rdrobust (2023).
Calonico, S., Cattaneo, M. D. & Titiunik, R. Optimal data-driven regression discontinuity plots. J. Am. Stat. Assoc. 110, 1753–1769 (2015).
Calonico, S., Cattaneo, M. D. & Farrell, M. H. On the effect of bias estimation on coverage accuracy in nonparametric inference. J. Am. Stat. Assoc. 113, 767–779 (2018).
Zeileis, A. & Hothorn, T. Diagnostic checking in regression relationships. R News 2, 7–10 (2002).
Cameron, A. C., Gelbach, J. B. & Miller, D. L. Robust inference with multiway clustering. J. Bus. Econ. Stat. 29, 238–249 (2011).
Zeileis, A. Econometric computing with HC and HAC covariance matrix estimators. J. Stat. Softw. https://doi.org/10.18637/jss.v011.i10 (2004).
Eckles, D., Karrer, B. & Johan, U. Design and analysis of experiments in networks: reducing bias from interference. J. Causal Inference https://doi.org/10.1515/jci-2015-0021 (2016).
Acknowledgements
The authors thank N. Grinberg, L. Friedland and K. Joseph for earlier technical work on the development of the Twitter dataset. Earlier versions of this paper were presented at the Social Media Analysis Workshop, UC Riverside, 26 August 2022; at the Annual Meeting of the American Political Science Association, 17 September 2022; and at the Center for Social Media and Politics, NYU, 23 April 2021. Special thanks go to A. Guess for suggesting the DID analysis. D.M.J.L. acknowledges support from the William & Flora Hewlett Foundation and the Volkswagen Foundation. S.D.M. was supported by the John S. and James L. Knight Foundation through a grant to the Institute for Data, Democracy & Politics at the George Washington University.
Author information
Authors and Affiliations
Contributions
The order of author listed here does not indicate level of contribution. Conceptualization of theory and research design: S.D.M., D.M.J.L., D.F., K.M.E. and J.G. Data curation: S.D.M. and J.G. Methodology: D.F. Visualization: D.F. Funding acquisition: D.M.J.L. Project administration: K.M.E., S.D.M. and D.M.J.L. Writing, original draft: K.M.E. and D.M.J.L. Writing, review and editing: K.M.E., D.F., S.D.M., D.M.J.L. and J.G.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Peer review
Peer review information
Nature thanks Jason Reifler and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Peer review reports are available.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Extended data figures and tables
Extended Data Fig. 1 Replication of the DID results varying the number of deplatformed accounts.
DID estimates where the intervention depends on the number of deplatformed users that were followed by the not-deplatformed misinformation sharers. Results are two-way fixed effect point estimates (dots) and 95% confidence intervals (bars) of the difference-in-differences for all activity levels combined. Estimates use ordinary least squares with clustered standard errors at user-level. The Figure shows results including and excluding Trump followers (color code). The x-axis shows the minimum number of deplatformed accounts the user followed from at least one (1+) to at least ten (10+). Total sample sizes for each dosage level: Follow Trump (No): 1: 625,865; 2: 538,460; 3: 495,723; 4: 470,380; 5: 451,468; 6: 437,574; 7: 426,772; 8: 417,200; 9: 408,672; 10: 401,467; Follow Trump (Yes): 1: 688,174; 2: 570,637; 3: 514,352; 4: 481,684; 5: 460,676; 6: 444,656; 7: 432,659; 8: 421,924; 9: 413,241; 10: 405,766.
Extended Data Fig. 2 SRD results for total (bottom row) and average (top row) misinformation tweets and retweets, for deplatformed and not-deplatformed users.
Sample size includes 546 observations (days) on average across groups (x-axis), 404 before and 136 after. The effective number of observations is 64.31 days before and after on average. The estimation excludes data between Jan 6 (cutoff point) and 12 (included). January 6th is the score value 0, and January 12th the score value 1. Optimal bandwidth of 32.6 days with triangular kernel and order-one polynomial. Bars indicate 95% robust bias-corrected confidence intervals.
Extended Data Fig. 3 Time series of the daily mean of non-misinformation URL sharing.
Degree five polynomial regression (fitted line) before and after the deplatforming intervention, separated by subgroup (panel rows), for liberal-slant news (right column), and conservative-slant news (left column) sharing activity. Shaded area around the fitted line is the 95% confidence interval of the fitted values. As a placebo test we evaluate the effect of the intervention on sharing non-fake news for each of our subgroups. Since sharing non-misinformation does not violate Twitter’s Civic Integrity policy – irrespective of the ideological slant of the news – we do not expect the intervention to have an impact on this form of Twitter engagement; see SI for how we identify liberal and conservative slant of these domains from ref. 52. Among the subgroups, users typically did not change their sharing of liberal or conservative non-fake news. Taking these results alongside those in Fig. 2 implies that these subgroups of users did not substitute non-misinformation conservative news sharing during and after the insurrection in place of misinformation.
Extended Data Fig. 4 Time series of misinformation tweets and retweets (panel columns), separately for high, medium and low activity users (panel rows).
Fitted straight lines describe a linear regression fitted using ordinary least squares of daily total misinformation retweeted standardized (y-axis) on days (x-axis) before January 6th and after January 12th. Shaded areas around the fitted line are 95% confidence intervals.
Extended Data Fig. 5 Replicates Fig. 5 but with adjustment covariates.
Corresponding regression tables are Supplementary Information Tables 1 to 3. Two-way fixed effect point estimates (dots) and 95% confidence intervals (bars) of the difference-in-differences for high, moderate, and low activity users, as well as all these levels combined (x-axis). P-values (stars) are from two-sided t-tests based on ordinary least squares estimates with clustered standard errors at user-level. Estimates compare followers (treated group) and not-followers (reference group) of deplatformed users after January 12th (post-treatment period) and before January 6th (pre-treatment period). No multiple test correction was used. See Supplementary Information Tables 1–3 for exact values with all activity level users combined. Total sample sizes of not-followers (reference) and Trump-only followers: combined: 306,089, high: 53,962, moderate: 219,375, low: 32,003; Followers: combined: 662,216, high: 156,941, moderate: 449,560, low: 53,442; Followers (4+): combined: 463,176, high: 115,264, moderate: 302,907, low: 43,218.
Extended Data Fig. 6 Placebo test of SRD results for total (bottom row) and average (top row) shopping and sports tweets and retweets at the deplatforming intervention, among those not deplatformed.
Sample size includes 545 observations (days), 404 before the intervention and 141 after. Optimal bandwidth of 843.6 days with triangular kernel and order-one polynomial. Cutoff points on January 6th (score 0) and January 12th (score 1). Bars indicate 95% robust bias-corrected confidence intervals. These are placebo tests since tweets about sports and shoppings should not be affected by the insurrection or deplatforming.
Extended Data Fig. 7 Placebo test of SRD results for total (bottom row) and average (top row) misinformation tweets and retweets using December 20th as an arbitrary cutoff point.
Sample size includes 551 observations (days), 387 before the intervention and 164 after. Optimal bandwidth of 37.2 days with triangular kernel and order-one polynomial. Bars indicate 95% robust bias-corrected confidence intervals about the SRD coefficients. This is a placebo test of the intervention period.
Extended Data Fig. 8 Placebo test of SRD results for total (bottom row) and average (top row) misinformation tweets and retweets using January 18th as a cutoff point.
The parameters are very similar to Extended Data Fig. 7.
Supplementary information
Supplementary Information
Supplementary Figs. 1–5 provide descriptive information about our subgroups, a replication of the panel data using the Decahose, and robustness analyses for the SRD. Supplementary Tables 1–5 show full parameter estimates for the DID models, summary statistics for follower type and activity level, and P values for the DID analyses under different multiple comparisons corrections.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
McCabe, S.D., Ferrari, D., Green, J. et al. Post-January 6th deplatforming reduced the reach of misinformation on Twitter. Nature 630, 132–140 (2024). https://doi.org/10.1038/s41586-024-07524-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/s41586-024-07524-8
Comments
By submitting a comment you agree to abide by our Terms and Community Guidelines. If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.