Measuring and Explaining Political Sophistication through Textual Complexity

View Researcher's Other Codes

Disclaimer: The provided code links for this paper are external links. Science Nest has no responsibility for the accuracy, legality or content of these links. Also, by downloading this code(s), you agree to comply with the terms of use as set out by the author(s) of the code(s).

Authors Kenneth Benoit, Kevin M. Munger, Arthur Spirling
Paper Category
Paper Abstract Political scientists lack domain-specific measures for the purpose of measuring the sophistication of political communication. We systematically review the shortcomings of existing approaches, before developing a new and better method along with software tools to apply it. We use crowdsourcing to perform thousands of pairwise comparisons of text snippets and incorporate these results into a statistical model of sophistication. This includes previously excluded features such as parts of speech and a measure of word rarity derived from dynamic term frequencies in the Google Books data set. Our technique not only shows which features are appropriate to the political domain and how, but also provides a measure easily applied and rescaled to political texts in a way that facilitates probabilistic comparisons. We reanalyze the State of the Union corpus to demonstrate how conclusions differ when using our improved approach, including the ability to compare complexity as a function of covariates. Replication Materials: The data, code, and any additional materials required to replicate all analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network, at: Akey concern in the study of politics is how the nature of political communication has changed. At the same time that the challenges of governing have grown in complexity, the sophistication of political speech, by many measures, appears to have declined. Typically as part of a broader discussion concerning “dumbing down” (Gatto 2002), scholars have applied measures of textual complexity from educational fields to find that the sophistication of political language has steadily decreased over the past 200 years (e.g., Lim 2008). Such concerns are echoed in popular presentations, and it is not uncommon to see media analysis assessing political speeches in terms Kenneth Benoit is Professor of Computational Social Science, London School of Economics, Houghton Street, London WC2A 2AE, UK ( Kevin Munger Affiliate Assistant Professor of Political Science and Social Data Analytics, Pennsylvania State University, 203 Pond Laboratory, State College, PA 16801 ( Arthur Spirling is Associate Professor of Politics and Data Science, New York University, 19 West 4th Street, New York, NY 10012 ( This research was partly supported by the European Research Council grant ERC-2011-StG283794-QUANTESS. The authors are grateful to audiences at the Midwest Political Science Association annual meeting, at the European Political Science Association annual meeting, and at the New Directions in Analyzing Text as Data meeting. Jacob Montgomery provided very helpful feedback on an earlier draft. Three anonymous referees and the editor at the American Journal of Political Science provided excellent comments that improved the paper considerably. For instance, see “Trump Speaks at Fourth-Grade Level, Lowest of Last 15 U.S. Presidents, New Analysis Finds,” Newsweek, January 8, 2018. See also “The State of Our Union Is . . . Dumber: How the Linguistic Standard of the Presidential Address Has Declined,” The Guardian, February 12, 2013. of the (purported lower) school grade level required to understand them.1 By contrast, and with more optimistic conclusions, other social science studies have used measures of textual complexity to link linguistic sophistication to outcomes, with a focus on the concrete benefits of clarity. Jansen (2011), for instance, studies the reading level of communications from four central banks, equating lower reading levels of bank communication with greater clarity, which they link to positive effects on the volatility of financial market returns. Likewise, Owens and Wedeking (2011) and Spriggs (1996) examine the complexity of American Journal of Political Science, Vol. 00, No. 00, xxxx 2019, Pp. 1–18 C © 2019 The Authors. American Journal of Political Science published by Wiley Periodicals, Inc. on behalf of Society for American Journal of Political Science DOI: 10.1111/ajps.12423 This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.
Date of publication 2017
Code Programming Language R

Copyright Researcher 2022