Should Researchers Trust the Mechanical Turk?

Mechanical Turk (MT) is an online labor marketplace run by Amazon in which large groups of online workers ("Turkers") can be recruited to carry out routine tasks that for whatever reason can’t be automated — tagging content in online videos for instance. (The service is named after a famous 18th-century fraud in which people were ...

By , a former associate editor at Foreign Policy.
Wikipedia
Wikipedia
Wikipedia

Mechanical Turk (MT) is an online labor marketplace run by Amazon in which large groups of online workers ("Turkers") can be recruited to carry out routine tasks that for whatever reason can't be automated -- tagging content in online videos for instance. (The service is named after a famous 18th-century fraud in which people were fooled into believing they were playing chess against a life-size supposedly automated "Turk" who was in fact controlled by a skilled human player.)

In recent years, MT has become a popular way for social scientists, particularly psychologists, to recruit test subjects. Turkers come cheap after all -- they're often paid as little as $1.50 an hour -- and are available in abundance. A number of the papers I've discussed on this site have relied in whole or in part on MT samples.

In a two-part blog post, Dan Kahan -- a professor of law and psychology at Yale University -- takes researchers to task for relying on Turk subjects in their research. Kahan discusses three primary flaws with Turk samples:

Mechanical Turk (MT) is an online labor marketplace run by Amazon in which large groups of online workers ("Turkers") can be recruited to carry out routine tasks that for whatever reason can’t be automated — tagging content in online videos for instance. (The service is named after a famous 18th-century fraud in which people were fooled into believing they were playing chess against a life-size supposedly automated "Turk" who was in fact controlled by a skilled human player.)

In recent years, MT has become a popular way for social scientists, particularly psychologists, to recruit test subjects. Turkers come cheap after all — they’re often paid as little as $1.50 an hour — and are available in abundance. A number of the papers I’ve discussed on this site have relied in whole or in part on MT samples.

In a two-part blog post, Dan Kahan — a professor of law and psychology at Yale University — takes researchers to task for relying on Turk subjects in their research. Kahan discusses three primary flaws with Turk samples:

1. Selection bias. Given the types of tasks performed by MT workers, there’s good reason to suspect subjects recruited via MT differ in material ways from the people in the world whose dispositions we are interested in measuring, particularly conservative males.

2. Prior, repeated exposure to study measures. Many MT workers have participated multiple times in studies that use performance-based measures of cognition and have discussed among themselves what the answers are. Their scores are thus not valid.

3. MT subjects misrepresent their nationality. Some fraction of the MT work force participating in studies that are limited to "U.S. residents only" aren’t in fact U.S. residents, thereby defeating inferences about how psychological dynamics distinctive of U.S. citizens of diverse ideologies operate.

On the first point, Kahan notes that MT workers are 62 percent female, only 5 percent African-American (the general U.S. population is 12 percent), and 53 percent self-identified liberals (only 20 percent of the general population.)

The problem of representativeness in test samples isn’t limited to MT of course. A 2011 University of British Columbia Study noted that "in the top international journals in six fields of psychology from 2003 to 2007, 68 percent of subjects came from the United States — and a whopping 96 percent from Western, industrialized countries. In one journal, 67 percent of American subjects and 80 percent of non-American subjects were undergraduates in psychology courses." This means that a good deal of what we know about the human brain comes from research on so-called WEIRD — Western, educated, industrialized, rich, and democratic societies — test subjects.

Still, if Turk-recruited subjects are essentially becoming professional test-takers, misrepresenting their qualifications and even their nationality in order to be eligible for studies, that’s a bigger problem than making generalizations based on the responses of UC undergrads.

And the bad press continues for psych research.

Joshua Keating was an associate editor at Foreign Policy. Twitter: @joshuakeating

More from Foreign Policy

An illustration shows George Kennan, the father of Cold War containment strategy.
An illustration shows George Kennan, the father of Cold War containment strategy.

Is Cold War Inevitable?

A new biography of George Kennan, the father of containment, raises questions about whether the old Cold War—and the emerging one with China—could have been avoided.

U.S. President Joe Biden speaks on the DISCLOSE Act.
U.S. President Joe Biden speaks on the DISCLOSE Act.

So You Want to Buy an Ambassadorship

The United States is the only Western government that routinely rewards mega-donors with top diplomatic posts.

Chinese President Xi jinping  toasts the guests during a banquet marking the 70th anniversary of the founding of the People's Republic of China on September 30, 2019 in Beijing, China.
Chinese President Xi jinping toasts the guests during a banquet marking the 70th anniversary of the founding of the People's Republic of China on September 30, 2019 in Beijing, China.

Can China Pull Off Its Charm Offensive?

Why Beijing’s foreign-policy reset will—or won’t—work out.

Turkish Defense Minister Hulusi Akar chairs a meeting in Ankara, Turkey on Nov. 21, 2022.
Turkish Defense Minister Hulusi Akar chairs a meeting in Ankara, Turkey on Nov. 21, 2022.

Turkey’s Problem Isn’t Sweden. It’s the United States.

Erdogan has focused on Stockholm’s stance toward Kurdish exile groups, but Ankara’s real demand is the end of U.S. support for Kurds in Syria.