• Graduate program
  • Research
  • News
  • Events
    • Summer School
      • Climate Change
      • Gender in Society
      • Inequalities in Health and Healthcare
      • Business Data Science Summer School Program
      • Receive updates
    • Events Calendar
    • Events Archive
    • Tinbergen Institute Lectures
    • Conference: Consumer Search and Markets
    • Annual Tinbergen Institute Conference
  • Summer School
    • Climate Change
    • Gender in Society
    • Inequalities in Health and Healthcare
    • Business Data Science Summer School Program
    • Receive updates
  • Alumni
  • Magazine
Home | Events Archive | We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments
Seminar

We Need to Talk about Mechanical Turk: What 22,989 Hypothesis Tests Tell Us about Publication Bias and p-Hacking in Online Experiments


  • Series
  • Speaker(s)
    Abel Brodeur (University of Ottawa, Canada)
  • Field
    Empirical Microeconomics
  • Location
    Online
  • Date and time

    September 23, 2022
    15:30 - 16:45

Amazon Mechanical Turk is a very widely-used tool in business and economics research, but how trustworthy are results from well-published studies that use it? Analyzing the universe of hypotheses tested on the platform and published in leading journals between 2010 and 2020 we find evidence of widespread p-hacking, publication bias and over-reliance on results from plausibly under-powered studies. Even ignoring questions arising from the characteristics and behaviors of study recruits, the conduct of the research community itself erode substantially the credibility of these studies' conclusions. The extent of the problems vary across the business, economics, management and marketing research fields (with marketing especially afflicted). The problems are not getting better over time and are much more prevalent than in a comparison set of non-online experiments. We explore correlates of increased credibility.

If you wish to attend the seminar or would like to have a bilateral with the speaker, please contact the seminar organizers Anne Boring (boring@ese.eur.nl) or Robert Dur (dur@ese.eur.nl).