The Shape of and Solutions to the MTurk Quality Crisis

View Researcher's Other Codes

Disclaimer: The provided code links for this paper are external links. Science Nest has no responsibility for the accuracy, legality or content of these links. Also, by downloading this code(s), you agree to comply with the terms of use as set out by the author(s) of the code(s).

Authors Ryan Kennedy, Scott J. Clifford, Tyler J. Burleigh, Ryan M Jewell, Philip D. Waggoner
Paper Category
Paper Abstract Amazon’s Mechanical Turk (MTurk) is widely used for data collection, however, researchers recently noticed a decline in data quality, stemming from the use of Virtual Private Servers (VPSs) to fraudulently gain access to surveys restricted to US residents. Unfortunately, we know little about the scale and consequence of this fraud, and tools for social scientists to detect and prevent this fraud are underdeveloped. Analyzing 38 studies conducted on MTurk since 2013, we demonstrate that this problem has recently spiked, but is not new. Two new studies show that these respondents provide particularly low-quality data. We provide three solutions: software to identify fraud in existing datasets, an easy-to-use web application based on this software, and a method for blocking fraudulent respondents in Qualtrics surveys. We demonstrate the effectiveness of the screening procedure in a third study. Our results suggest that these fraudulent respondents provide unusually low-quality data, but can be easily identified and screened out.
Date of publication 2018
Code Programming Language R

Copyright Researcher 2022