What Do Replication Failures Tell Us About Priming?

Despite extensive evidence for priming in general and behavior priming in particular, there has emerged in recent years a backlash against this research, based largely on the publication of a number of replication failures of highly-cited priming studies.[1] Some observers have suggested these results are symptomatic of a deeper replication crisis (sometimes called a reproducibility crisis) that is variously described as implicating behavior priming research in particular, priming research in general, social psychology and psychology in general, all the social sciences, or indeed, the entire enterprise of science.[2]

Replication is one of the most important tools of scientific inquiry. After an experiment or study is completed, the original authors or other researchers often try to repeat it to see if they get the same results. The more times the same results are achieved, that stronger and more reliable the conclusions are taken to be. Conversely, if the study fails to replicate, this can be a valuable result as well. It means researchers have more to learn about the original study; either about the causes or mechanisms that produced its results, or about some hidden aspect of the original procedure that inadvertently impacted the outcome.

Failed replications have become a significant issue in psychology. In a massive, multi-lab replication effort launched in 2012 and completed in 2015, researchers attempted to replicate 100 studies—many of which focused on priming—from three psychology journals. They found that while 97% of the original studies reported statistically significant results, only 36% of the replications did. Overall, the effect sizes[3] found in the replications were only about half as large as in the original studies, implying that the original studies may have seriously overestimated the magnitudes of the effects they were measuring.[4]

Naturally, practitioners and commentators want to know why replication failures seem to occur with such frequency, and what this means for the scientific status of the underlying mechanisms and processes being studied.

One source of the problem is prevalent throughout academic science: the norms and practices of scientific publishing. Scientific journals have a natural bias to publish positive results. Positive results are more interesting, more newsworthy, and presumably more prestigious to publish in your journal than negative findings. So academic journals in all fields tend to select the most positive studies, with the largest effect sizes, for publication. That bias, in turn, affects the submission strategies of researchers, all of whom have strong career incentives to get published. Researchers therefore tend to submit their positive findings to journals and put their negative findings aside. This leads to what is called the file drawer problem in which negative results do not get reported, resulting in a skewing of published findings in an artificially positive direction.[5] That some of these positive findings should fail to replicate, given they come from a positively-biased sample of all research conducted, should not be completely unexpected.

Is Behavior Priming Real?

The basic idea behind priming—that exposure to an incidental stimulus can unconsciously influence a person’s subsequent memory, judgment, and choices—has been observed, replicated, and accepted as a basic building block of human cognition at least since the early 1950s.[6] The question is whether behavior—actual physical actions—can be added to this list of things priming can influence.

A large-scale meta-analysis of behavior priming research published in 2016 provided a positive answer. Led by University of Pennsylvania psychologist Evan Weingarten, the analysis examined the results of over 350 tests from 133 studies in which incidentally-presented words were used to prime actions of various kinds. Effect sizes were calculated for behavior differences between primed and non-primed (control group) participants across all 350 tests. Statistical tests revealed a significant overall positive effect of priming on behavior, but of a relatively small size. The researchers concluded that behavior priming was indeed real, but smaller in its average impact than had been reported in some highly-cited early studies. An additional test for publication bias found some impact of publication vs. non-publication on effect sizes, but not enough to invalidate the overall findings.[7]

How Should Marketers Think about Behavior Priming?

Nonscientists often make the mistake of believing that an individual “finding” in a single academic paper is the fundamental unit of scientific discovery. That is, they read a paper (or read about a paper), they see what it claims to show, they see it is reported to have a 19-in-20 chance of being true (p=.05), so they assume it must be both true and generalizable. This misconception is reinforced by popular science news and articles, in which a new finding—especially one that is surprising or counterintuitive—is described as a “new truth” that can be directly applied to real-world situations, often situations involving marketing or consumer persuasion.

As the replication crisis shows, there are many biases in the scientific enterprise. As discussed at length in Intuitive Marketing,[8] there are also many reasons why a single finding might not replicate.

The power of science emerges not from any single study, but from the slow cumulation and convergence of many findings, often over hundreds of studies and decades of research.

The real fundamental unit of discovery in science is the evolving body of knowledge produced by a research community of scholars and practitioners who focus their work and careers on understanding a slice of reality or human behavior.[9]

Marketers and consumers need to look at the body of knowledge around priming as a whole, whether labeled social priming, motivational priming, goal priming, or behavior priming. What they should take away from this body of knowledge, especially following the intensive scrutiny it received as a part of the replication crisis, can be summarized in a few bullet points:

• Priming is a real phenomenon. Failure to replicate individual studies can be traced to many causes, none of which nullify the reality of the underlying psychological mechanism.
• Human thoughts and behavior are influenced by peripheral cues in our environments and mental representations in our minds, both consciously and unconsciously.
• Effect sizes of behavior priming are not as large as many early studies reported. Priming regularly produces predictable results in highly controlled laboratory environments, but its effects in the real world, where people are exposed to large numbers of competing primes at every moment, are much less certain and predictable.
• The problem of competing primes is likely to be even more troublesome in shopping contexts and cluttered advertising environments, where multiple marketing messages and cues are competing against each other to influence consumers, both consciously and unconsciously.
• The priming effects of marketing on consumer behavior are unlikely to be universal and reliably predictable, because they can be moderated by different contextual features as well as different beliefs, goals, and attitudes in the minds of consumers.
• Motivational relevance is a key predictor of priming effectiveness. If consumers are not already motivated to pursue a particular goal or outcome, priming is unlikely to have much of an effect on their subsequent thoughts and behavior.
• This means that priming is unlikely to be effective as a persuasion technique in marketing—assuming the marketer’s purpose is to instill a new goal to consider, purchase, or consume a product or service. Priming is more likely to be effective when it acts as a trigger associating a product or service with an existing goal or aspiration, one that is already active in the mind of a consumer.

The replication crisis in behavior priming research is an example of science working as it is supposed to work, with a body of knowledge self-correcting both its findings and its methods in light of new information emerging over time. The message to marketers and consumers is fourfold:

• Draw lessons not from single studies and single results, but from cumulation and convergence across a body of studies.
• Remember that scientific results are always tentative, never certain.
• Keep in mind that what looks too good to be true probably is.
• Don’t throw out the baby with the bath water.

(This post is a slightly condensed excerpt from Intuitive Marketing: What Marketers Can Learn from Brain Science, pp. 212-223.)

Image Creator: Biljana Cvetanovic, Getty Images/iStockphoto

References

[1] For example, Doyen, Stéphane, et al. “Behavioral priming: it’s all in the mind, but whose mind?” PloS one 7.1 (2012): e29081; Harris, Christine R., et al. “Two failures to replicate high-performance-goal priming effects.” PloS one 8.8 (2013): e72467.

[2] With regard to science in general, an often-cited assessment is provided by Ioannidis, John PA. “Why most published research findings are false.” PLoS medicine 2.8 (2005): e124.

[3] See Coe, Robert. “It’s the effect size, stupid: What effect size is and why it is important.” (2002), available online at http://bit.ly/2EKMNEx:

‘Effect size’ is simply a way of quantifying the size of the difference between two groups. It is easy to calculate, readily understood and can be applied to any measured outcome …. It is particularly valuable for quantifying the effectiveness of a particular intervention, relative to some comparison.

See also Genco et al., Neuromarketing for Dummies. Wiley, 2013, p. 312.

[4] Open Science Collaboration. “Estimating the reproducibility of psychological science.” Science 349.6251 (2015): aac4716.

[5] Rosenthal, Robert. “The file drawer problem and tolerance for null results.” Psychological bulletin 86.3 (1979): 638.

[6] Bargh, John A. “The historical origins of priming as the preparation of behavioral responses: Unconscious carryover and contextual influences of real-world importance.” Social Cognition 32. Supplement (2014): 209-224.

[7] Weingarten, Evan, et al. “From primed concepts to action: A meta-analysis of the behavioral effects of incidentally presented words.” Psychological Bulletin 142.5 (2016): 472.

[8] Genco, Stephen. Intuitive Marketing: What Marketers Can Learn from Brain Science. Intuitive Consumer Insights, 2019, pp. 212-223.

[9] For a detailed discussion and evolutionary rationale for this view of science, see Toulmin, Stephen. Human Understanding: The Collective Use and Evolution of Concepts. Princeton University Press, 1975.

About the Author: Steve Genco

Steve is a writer, speaker, researcher, and marketing consultant. He is author of Intuitive Marketing (2019), a study of persuasion and influence in marketing theory and practice, and co-author of Neuromarketing for Dummies (2013), a comprehensive overview of neuromarketing science, applications, methodologies, and ethics. He is Managing Partner at Intuitive Consumer Insights, where he focuses on marketing education and consulting.