> "For academic researchers: worker IDs may have to be treated as personally identifiable information. For example, publishing worker IDs online in public data sets may be a violation of worker privacy, and counter to the requirements of the researcher’s institutional review board."
This is why we created http://SocialSci.com for use by academic researchers: We have better control over PII outside the Amazon ecosystem, and access to more data we can take our own steps to protect or de-identify, enabling higher quality data collection.
In general academic research done on Mechanical Turk has a ton of issues, from response quality to verifiability of responses. Frankly people have no incentive to remain authentic across surveys, and we've seen cases of people just taking on identities or just faking answers, which isn't reliably detectable. The only way we've been able to overcome this is by tracking users' response consistency over time, and keeping them anonymous in the process. It's proved to be a much more reliable method, and at this point based on our experience we can't recommend MTurk for valid academic research.
You will always have these issues. If people want to cheat the system they can create fake accounts, administrate profiles for family members and so on. Isolated surveys, no matter if qualitative or quantitative questions, are quite limited in their implications as long as there are incentives to give dishonest responses and no other ways to evaluate the data in the context of practice exist.
Developer at SocialSci here. You're absolutely correct, the incentive for gaming the system is always there. The difference is that we also incentivize consistency and are very proactive about tracking down fake accounts through various means. You're right that it's a difficult problem, but while Amazon turns a blind eye to such abuse, our system is designed from the start to both disincentivize and prevent it, and in the meantime encourage good consistent high-quality responses over time.
We don't claim to entirely prevent fraud, but we're very confident that we raise the barrier and reduce the rate significantly.
It seems like academic researchers would already have been reluctant to publish worker IDs unless they had some assurance from Amazon they were anonymized somehow. Assuming it is anonymized is pretty bad practice.
This is why we created http://SocialSci.com for use by academic researchers: We have better control over PII outside the Amazon ecosystem, and access to more data we can take our own steps to protect or de-identify, enabling higher quality data collection.
In general academic research done on Mechanical Turk has a ton of issues, from response quality to verifiability of responses. Frankly people have no incentive to remain authentic across surveys, and we've seen cases of people just taking on identities or just faking answers, which isn't reliably detectable. The only way we've been able to overcome this is by tracking users' response consistency over time, and keeping them anonymous in the process. It's proved to be a much more reliable method, and at this point based on our experience we can't recommend MTurk for valid academic research.