ABSTRACT
We identify a pervasive, yet previously undocumented threat to the reliability of MTurk data—and discuss how this issue is symptomatic of opportunities and incentives that facilitate fraudulent behavior within online recruitment platforms. In doing so, we explain how IP addresses were never intended to identify individuals and are likely insufficient to identify and mitigate emergent risks around data integrity. We discuss MTurk samples for two studies that include alarming proportions of participants who circumvent an entire set of conventional sample screening methods—and provide disturbingly low-quality responses. These “bad actors” exploited inherent limitations of IP screening procedures by using virtual private servers (VPS) that concealed the IP address of their local devices. While service providers now help target this abuse, the underlying limitations of IP screening procedures remain. Our findings emphasize the importance of continued diligence within the research community to identify and mitigate evolving threats to data integrity.
