How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It

View Researcher's Other Codes

Disclaimer: The provided code links for this paper are external links. Science Nest has no responsibility for the accuracy, legality or content of these links. Also, by downloading this code(s), you agree to comply with the terms of use as set out by the author(s) of the code(s).

Authors Jacob Michael Montgomery, Brendan Nyhan, Michelle Torres
Journal/Conference Name AMERICAN JOURNAL OF POLITICAL SCIENCE
Paper Category
Paper Abstract In principle, experiments o↵er a straightforward method for social scientists to accurately estimate causal e↵ects. However, scholars often unwittingly distort treatment e↵ect estimates by conditioning on variables that could be a↵ected by their experimental manipulation. Typical examples include controlling for post-treatment variables in statistical models, eliminating observations based on post-treatment criteria, or subsetting the data based on post-treatment variables. Though these modeling choices are intended to address common problems encountered when conducting experiments, they can bias estimates of causal e↵ects. Moreover, problems associated with conditioning on post-treatment variables remain largely unrecognized in the field, which we show frequently publishes experimental studies using these practices in our discipline’s most prestigious journals. We demonstrate the severity of experimental post-treatment bias analytically and document the magnitude of the potential distortions it induces using visualizations and reanalyses of real-world data. We conclude by providing applied researchers with recommendations for best practice. ⇤Authors are listed in alphabetical order. The data, code, and any additional materials required to replicate all analyses in this article are available on the American Journal of Political Science Dataverse within the Harvard Dataverse Network at http://dx.doi.org/10.7910/DVN/EZSJ1S. We thank David Broockman, Daniel Butler, Eric S. Dickson, Sanford Gordon, and Gregory Huber for sharing replication data and Ryden Butler, Lindsay Keare, Jake McNichol, Ramtin Rahmani, Rebecca Rodriguez, Erin Rossiter, and Caroline Sohr for research assistance. We are also grateful to Dan Butler, Jake Bowers, Scott Cli↵ord, Eric S. Dickson, D.J. Flynn, Sanford Gordon, Gregory Huber, Jonathan Ladd, David Nickerson, Efrén O. Pérez, Julian Schuessler, Molly Roberts, and three anonymous reviewers for helpful comments. All errors are our own.
Date of publication 2018
Code Programming Language R
Comment

Copyright Researcher 2022