Be smart about combining self-reporting and nonconscious measures in media research

Combining self-reporting and nonconscious measures is something everybody agrees is A GOOD THING, but nobody seems to say much about how to do it, or how not to do it. I recently came across a terrific exception — Robert Potter and Paul Bolls’ 2011 book with the mouthful title Psychophysiological Measurement and Meaning: Cognitive and Emotional Processing of Media, especially their detailed Chapter 7, “Connecting psychophysiology to other measures of mediated message processing.”

This book is about both doing the right thing and doing it right. It is written as an introduction to the budding psychophysiologist, but for the rest of us it is a dense and detailed read, full of good science and practical advice for measuring and interpreting signals from the body and brain as people are exposed to media like TV programs, movies, online videos, and advertising.

Potter and Bolls are veterans of the legendary media program established by Annie Lang at Indiana University, and they bring decades of experience to this book. For neuromarketers and clients of neuromarketing, the book serves three important purposes: (1) it describes a wide range of practical measurements that relate to people’s responses to media (including marketing and advertising messages), (2) it summarizes large bodies of academic research that validate these measures, and (3) it shows how easy it is to mess up collecting those measures. The third purpose may be the most practically relevant. After reading all the ways psychophysiological measurements can be compromised, we should all have a renewed appreciation for the professionalism and experience needed to obtain accurate nonconscious measures. Psychophysiology is not for do-it-yourself’ers.

This rather long post provides a summary of what Potter and Bolls have to say about combining self-reporting and nonconscious measures. The former include things like surveys and interviews, while the latter includes things like skin conductance, muscle movement, heart rate, and brain waves. Potter and Bolls’ insights take us well beyond the usual “compare and contrast” pablum commonly offered in discussions about how to bring these two approaches together.

The first key point, and one that many miss, is that these measures are not directly comparable and should not be seen as competing with each other. I have also criticized this “which is better?” perspective elsewhere, but Potter and Bolls provide a much richer and more detailed perspective:

the dependent variables of interest in media psychology research — attention, emotion, memory, attitudes, decision-making, etc. — emerge from the operation of multiple dynamic, embodied mental processes that ultimately yield numerous forms of meaningful experiences reaching varying levels of consciousness. (Kindle Locations 3843-3845)

Traditional self-reported outcome variables do not stand on their own:

‘outcome’ measures — representative of beliefs, attitudes, feelings, and behaviors — should not be viewed as indexing stable and predictable ‘effects’ of media exposure. The media psychology researcher should rather view self-report and behavioral measures of so-called ‘media effects’ as indexing psychological states that dynamically interact with mental processes as well as other psychological states across time in a broad complex social environment that includes media use. (Kindle Locations 3899-3902)

The key idea here is dynamic interaction. Self-reports result from an interaction with nonconscious responses. They cannot be fully understood in isolation from them. As a consequence, there is no reason to expect the two kinds of measures to be correlated:

Researchers … can be tempted to limit their understanding of the relationship between psychophysiological and self-report measures to an analysis of the degree to which they are correlated. We would strongly caution against such a mindset because when these two types of measures are found to not be significantly correlated, an easy default is to assume that one of the measures must be invalid. (Kindle Locations 3985-3989)

Three ways to combine self-report measures with biometrics

Potter and Bolls recommend three general ways self-report measures can be successfully combined with “psychophysiological measures of embodied mediated message processing” (let’s just call these “biometrics” to save a little space):

1. Index psychological states related to the embodied mental processes biometrics index.

This approach focuses on combining established, self-report psychometric scales measuring emotional feelings and attention allocation with biometric responses to media. For example, in studying responses to emotional pictures, a visual self-reporting emotional response metric called the Self-Assessment Mannequin (SAM) scale (see Figure 1) has been found to correlate positively with both skin conductance for arousal and EMG for valence.

Figure 1. SAM images for measuring emotional valence (top) and arousal.

Combining SAM and biometrics has been used to study responses to a wide variety of media, including:

• the visual complexity of websites
• negative political advertising
• avatar choice in multi-player computer games
• television ads
• structural complexity of radio messages

Reliable scales have also been developed for measuring self-reports of attention and cognitive load when consuming media. These scales provide insights into how individuals perceive their own mental efforts devoted to perceiving, attending to, and making sense of media content. By combining these self assessments with biometric measures, researchers have documented at least one finding of critical importance to neuromarketing — human beings have only limited awareness of how they actually allocate and target their attention.

2. Collect self-report measures of psychological states that are believed to either moderate or emerge from variation in biometric results.

Prior conscious states impact biometric responses to media consumption and emerge from that consumption. Researchers has found, for example, that conscious expectations can significantly alter emotional and attentional responses to media, and that conscious emotional states like moods, goals, or aggression are important consequence of media consumption.

These consciously-experienced before-and-after states can be measures with a wealth of validated scales available in the psychological literature, Here is another important implication for neuromarketers who claim they don’t need conscious measures to understand responses to media. Adding validated psychological scales to measure before and after effoect of media consumption can provide a much richer understanding of the media consumption experience than can be derived from biometric markers alone.

3. Use self-report psychological scales to index significant individual differences that might influence biometric results.

The role of individual differences in explaining responses to all kinds of media is a topic that has been under-explored in neuromarketing. For-profit studies usually focus on comparing different stimuli (like two ads, two packages, etc.) but they usually assume the sample of respondents is homogeneous. But people vary considerably across many psychological dimensions that are relevant to media consumption, and these differences can only be measured by having subjects complete validated self-report scales. Some dimensions that have proven to produce variations in responses to media consumption include:

• sensation seeking
• need for cognition
• intensity of emotional responding
• motivation activation
• personality characteristics

As with combining conscious measures of expectations and consequences, validated scales, not individual opinion questions, are the recommended approach to collecting self-report responses.

Three alternatives to self-reporting via survey questions

The chapter also looks at alternative self-reporting techniques to capture the media consumption experiences more dynamically. Three methods are discussed, each with its own pros and cons. An important point is that all three of these methods are measuring conscious responses, even though they are sometimes misclassified as nonconscious response measures.

1. Continuous response measurement (CRM)

CRM is in essence a moment-by-moment electronic form of self-report measurement that captures the qualitative experience of message processing, usually using a handheld dial or slider to continuously report responses along some dimension (like-dislike, agree-disagree, etc.). It has been used extensively in advertising and political messaging research.

Potter and Bolls argue that CRM responses should not be seen as a surrogate for nonconscious responses. They are clearly conscious responses that are filtered through a participant’s ongoing introspective evaluation of their mental states. Accordingly, the authors recommend that CRM only be used to measure very simple psychological states, like level of interest or simple favorable/unfavorable feelings. They caution:

The more deliberative participants need to be in their introspection in order to report variation in the psychological state being measured, the less valid CRM data is likely to be. Researchers need to keep in mind that the task of having to consciously evaluate and report variation in their mental state while simultaneously watching a message takes away cognitive resources from processing the message itself. (Kindle Location 4384)

2. Thought listing

This self-reporting technique, often called the “think aloud” procedure, is a measure that gives researchers a glimpse into the mental contents of individuals’ specific thoughts, feelings, ideas, expectations, appraisals, or mental images. If has been used extensively to study mental processing of persuasive media messages like ads. Thoughts evoked by a persuasive message are cognitive responses that researchers believe are highly relevant to attitude change following exposure to a message.

Potter and Bolls recommend that thought listing only be used to capture the most general outputs of mental processing, and that it be employed as close to the actual consumption of media as possible, to avoid memory recall biases and other contamination effects. They caution against asking people to explain their thoughts, because “individuals cannot be expected to explain why or how specific mental contents may have been produced by a stimulus.” Further:

The mental contents captured through the thought listing technique describe the output of mental processes, or more specifically, reflect … emerging psychological states. It therefore seems best to give participants as general a prompt as possible in instructing them to recall and state any thoughts that may have been evoked by a message. (Kindle Locations 4453-4456)

3. Secondary task reaction time (STRT)

The final alternative self-reporting technique the authors discuss is really a behavioral measure. STRT measures reaction times to a secondary task, like pressing a button in response to a brief audio or visual cue, while participants are concentrating on a primary task, usually consuming some form of media. The basic idea is that reactions to the secondary task will vary based on the extent to which cognitive resources and attention are being allocated to the primary task.

In practice, using STRT as a self-reporting method in conjunction with biometric measures of nonconscious processes can suffer from an interaction between the conscious secondary task response and the nonconscious biometric responses. As with CRM, the act of engaging a motor response (pressing a button) can interfere with biometric responses when the two are collected simultaneously.

The STRT cue is a stimulus that will evoke specific psychophysiological responses in addition to patterns of physiological responding evoked by a message. These additional responses are noise in the data a media psychology researcher is typically most interested in — psychophysiological responses evoked by the message. The motor responses required to respond to the secondary task reaction time cue have the potential to introduce even more noise into psychophysiological measures. (Kindle Locations 4573-4576)

As a consequence, the authors note, there are few published studies that collect secondary task reaction times and psychophysiological measures simultaneously on the same messages.

Summing it all up

This long post still only skims the surface of Potter and Bolls’ in-depth discussion of combining conscious self-reporting measures and nonconscious biometric measures when studying media consumption. The key takeaways for neuromarketing researchers and clients are these:

• Don’t judge the validity of one type of measure by comparing it directly to the other. Whether they correlate is an empirical question, not an assumption.
• Self reporting questions are most likely to add to the value of biometrics when they are theoretically derived and when they are based on reliable, valid, and well-established psychological scales.
• Behavioral alternatives to self-reporting surveys have their place in conscious-nonconscious research, but their strengths and weaknesses should be understood and, as always, experimental designs and methodologies should be developed by experienced researchers who are familiar with all the pros and cons of different measures and combinations of approaches.

Image: Neurogadget, little brain added.

About the Author: Steve Genco

Steve is a writer, speaker, researcher, and marketing consultant. He is author of Intuitive Marketing (2019), a study of persuasion and influence in marketing theory and practice, and co-author of Neuromarketing for Dummies (2013), a comprehensive overview of neuromarketing science, applications, methodologies, and ethics. He is Managing Partner at Intuitive Consumer Insights, where he focuses on marketing education and consulting.