Be smart about combining self-reporting and nonconscious measures in media research
Combining self-reporting and nonconscious measures is something everybody agrees is A GOOD THING, but nobody seems to say much about how to do it, or how not to do it. I recently came across a terrific exception — Robert Potter and Paul Bolls’ 2011 book with the mouthful title Psychophysiological Measurement and Meaning: Cognitive and Emotional Processing of Media, especially their detailed Chapter 7, “Connecting psychophysiology to other measures of mediated message processing.”
This book is about both doing the right thing and doing it right. It is written as an introduction to the budding psychophysiologist, but for the rest of us it is a dense and detailed read, full of good science and practical advice for measuring and interpreting signals from the body and brain as people are exposed to media like TV programs, movies, online videos, and advertising.
Potter and Bolls are veterans of the legendary media program established by Annie Lang at Indiana University, and they bring decades of experience to this book. For neuromarketers and clients of neuromarketing, the book serves three important purposes: (1) it describes a wide range of practical measurements that relate to people’s responses to media (including marketing and advertising messages), (2) it summarizes large bodies of academic research that validate these measures, and (3) it shows how easy it is to mess up collecting those measures. The third purpose may be the most practically relevant. After reading all the ways psychophysiological measurements can be compromised, we should all have a renewed appreciation for the professionalism and experience needed to obtain accurate nonconscious measures. Psychophysiology is not for do-it-yourself’ers.
This rather long post provides a summary of what Potter and Bolls have to say about combining self-reporting and nonconscious measures. The former include things like surveys and interviews, while the latter includes things like skin conductance, muscle movement, heart rate, and brain waves. Potter and Bolls’ insights take us well beyond the usual “compare and contrast” pablum commonly offered in discussions about how to bring these two approaches together.
The first key point, and one that many miss, is that these measures are not directly comparable and should not be seen as competing with each other. I have also criticized this “which is better?” perspective elsewhere, but Potter and Bolls provide a much richer and more detailed perspective:
the dependent variables of interest in media psychology research — attention, emotion, memory, attitudes, decision-making, etc. — emerge from the operation of multiple dynamic, embodied mental processes that ultimately yield numerous forms of meaningful experiences reaching varying levels of consciousness. (Kindle Locations 3843-3845)
Traditional self-reported outcome variables do not stand on their own:
‘outcome’ measures — representative of beliefs, attitudes, feelings, and behaviors — should not be viewed as indexing stable and predictable ‘effects’ of media exposure. The media psychology researcher should rather view self-report and behavioral measures of so-called ‘media effects’ as indexing psychological states that dynamically interact with mental processes as well as other psychological states across time in a broad complex social environment that includes media use. (Kindle Locations 3899-3902)
The key idea here is dynamic interaction. Self-reports result from an interaction with nonconscious responses. They cannot be fully understood in isolation from them. As a consequence, there is no reason to expect the two kinds of measures to be correlated:
Researchers … can be tempted to limit their understanding of the relationship between psychophysiological and self-report measures to an analysis of the degree to which they are correlated. We would strongly caution against such a mindset because when these two types of measures are found to not be significantly correlated, an easy default is to assume that one of the measures must be invalid. (Kindle Locations 3985-3989)
Three ways to combine self-report measures with biometrics
Potter and Bolls recommend three general ways self-report measures can be successfully combined with “psychophysiological measures of embodied mediated message processing” (let’s just call these “biometrics” to save a little space):
1. Index psychological states related to the embodied mental processes biometrics index.
This approach focuses on combining established, self-report psychometric scales measuring emotional feelings and attention allocation with biometric responses to media. For example, in studying responses to emotional pictures, a visual self-reporting emotional response metric called the Self-Assessment Mannequin (SAM) scale (see Figure 1) has been found to correlate positively with both skin conductance for arousal and EMG for valence.
Combining SAM and biometrics has been used to study responses to a wide variety of media, including:
- the visual complexity of websites
- negative political advertising
- avatar choice in multi-player computer games
- television ads
- structural complexity of radio messages
Reliable scales have also been developed for measuring self-reports of attention and cognitive load when consuming media. These scales provide insights into how individuals perceive their own mental efforts devoted to perceiving, attending to, and making sense of media content. By combining these self assessments with biometric measures, researchers have documented at least one finding of critical importance to neuromarketing — human beings have only limited awareness of how they actually allocate and target their attention.
2. Collect self-report measures of psychological states that are believed to either moderate or emerge from variation in biometric results.
Prior conscious states impact biometric responses to media consumption and emerge from that consumption. Researchers has found, for example, that conscious expectations can significantly alter emotional and attentional responses to media, and that conscious emotional states like moods, goals, or aggression are important consequence of media consumption.
These consciously-experienced before-and-after states can be measures with a wealth of validated scales available in the psychological literature, Here is another important implication for neuromarketers who claim they don’t need conscious measures to understand responses to media. Adding validated psychological scales to measure before and after effoect of media consumption can provide a much richer understanding of the media consumption experience than can be derived from biometric markers alone.
3. Use self-report psychological scales to index significant individual differences that might influence biometric results.
The role of individual differences in explaining responses to all kinds of media is a topic that has been under-explored in neuromarketing. For-profit studies usually focus on comparing different stimuli (like two ads, two packages, etc.) but they usually assume the sample of respondents is homogeneous. But people vary considerably across many psychological dimensions that are relevant to media consumption, and these differences can only be measured by having subjects complete validated self-report scales. Some dimensions that have proven to produce variations in responses to media consumption include:
- sensation seeking
- need for cognition
- intensity of emotional responding
- motivation activation
- personality characteristics
As with combining conscious measures of expectations and consequences, validated scales, not individual opinion questions, are the recommended approach to collecting self-report responses.
Three alternatives to self-reporting via survey questions
The chapter also looks at alternative self-reporting techniques to capture the media consumption experiences more dynamically. Three methods are discussed, each with its own pros and cons. An important point is that all three of these methods are measuring conscious responses, even though they are sometimes misclassified as nonconscious response measures.
1. Continuous response measurement (CRM)
CRM is in essence a moment-by-moment electronic form of self-report measurement that captures the qualitative experience of message processing, usually using a handheld dial or slider to continuously report responses along some dimension (like-dislike, agree-disagree, etc.). It has been used extensively in advertising and political messaging research.
Potter and Bolls argue that CRM responses should not be seen as a surrogate for nonconscious responses. They are clearly conscious responses that are filtered through a participant’s ongoing introspective evaluation of their mental states. Accordingly, the authors recommend that CRM only be used to measure very simple psychological states, like level of interest or simple favorable/unfavorable feelings. They caution:
The more deliberative participants need to be in their introspection in order to report variation in the psychological state being measured, the less valid CRM data is likely to be. Researchers need to keep in mind that the task of having to consciously evaluate and report variation in their mental state while simultaneously watching a message takes away cognitive resources from processing the message itself. (Kindle Location 4384)
2. Thought listing
This self-reporting technique, often called the “think aloud” procedure, is a measure that gives researchers a glimpse into the mental contents of individuals’ specific thoughts, feelings, ideas, expectations, appraisals, or mental images. If has been used extensively to study mental processing of persuasive media messages like ads. Thoughts evoked by a persuasive message are cognitive responses that researchers believe are highly relevant to attitude change following exposure to a message.
Potter and Bolls recommend that thought listing only be used to capture the most general outputs of mental processing, and that it be employed as close to the actual consumption of media as possible, to avoid memory recall biases and other contamination effects. They caution against asking people to explain their thoughts, because “individuals cannot be expected to explain why or how specific mental contents may have been produced by a stimulus.” Further:
The mental contents captured through the thought listing technique describe the output of mental processes, or more specifically, reflect … emerging psychological states. It therefore seems best to give participants as general a prompt as possible in instructing them to recall and state any thoughts that may have been evoked by a message. (Kindle Locations 4453-4456)
3. Secondary task reaction time (STRT)
The final alternative self-reporting technique the authors discuss is really a behavioral measure. STRT measures reaction times to a secondary task, like pressing a button in response to a brief audio or visual cue, while participants are concentrating on a primary task, usually consuming some form of media. The basic idea is that reactions to the secondary task will vary based on the extent to which cognitive resources and attention are being allocated to the primary task.
In practice, using STRT as a self-reporting method in conjunction with biometric measures of nonconscious processes can suffer from an interaction between the conscious secondary task response and the nonconscious biometric responses. As with CRM, the act of engaging a motor response (pressing a button) can interfere with biometric responses when the two are collected simultaneously.
The STRT cue is a stimulus that will evoke specific psychophysiological responses in addition to patterns of physiological responding evoked by a message. These additional responses are noise in the data a media psychology researcher is typically most interested in — psychophysiological responses evoked by the message. The motor responses required to respond to the secondary task reaction time cue have the potential to introduce even more noise into psychophysiological measures. (Kindle Locations 4573-4576)
As a consequence, the authors note, there are few published studies that collect secondary task reaction times and psychophysiological measures simultaneously on the same messages.
Summing it all up
This long post still only skims the surface of Potter and Bolls’ in-depth discussion of combining conscious self-reporting measures and nonconscious biometric measures when studying media consumption. The key takeaways for neuromarketing researchers and clients are these:
- Don’t judge the validity of one type of measure by comparing it directly to the other. Whether they correlate is an empirical question, not an assumption.
- Self reporting questions are most likely to add to the value of biometrics when they are theoretically derived and when they are based on reliable, valid, and well-established psychological scales.
- Behavioral alternatives to self-reporting surveys have their place in conscious-nonconscious research, but their strengths and weaknesses should be understood and, as always, experimental designs and methodologies should be developed by experienced researchers who are familiar with all the pros and cons of different measures and combinations of approaches.
Image: Neurogadget, little brain added.
About the Author: Steve Genco

Hi Steve,
Thanks for writing this summary. We find Potter and Bolls to be a great resource for our biometric research, but as you say the text can be very detailed and dense unless you spend most of your days in the lab. The translation you’re providing of this complex information for the market is very important.
I would also offer here that, while these recommendations are very good and should be practiced, the chapter (and book) is largely focused on media processing research. As Potter and Bolls note psychophysiological research was developed to “better understand psychological phenomena” and “how bodily activity [relates] to cognitive and emotional states”. Once we have measures (biometric or otherwise) that represent those cognitive and emotional states experienced during media processing or other processing, there are separate questions on how those psychological “states” interact to produce consumer decisions and behavior.
Given that predicting consumer behavior is often our ultimate objective in #mrx, there are critical questions facing our industry on how System 1 and System 2 processing interacts to determine consumer choice and specifically how we can most effectively measure automatic associative processing (1) and more deliberate propositional processing (2). In running these kinds of studies in our subconscious lab, we typically are looking for measures that explain unique variance in some separate behavioral measure of interest (e.g. product sales data), to determine the relative role of automatic versus deliberate processing on consumer behavior.
These types of processes are very challenging to tease apart, since they naturally interact. However, this kind of applied research, detailing which measures are best are capturing the unique contribution of automatic versus deliberate processing, and how to effectively combine them, is what I see as a necessary next step as we move neuromarketing methods from the early adoption to early majority stage.
I know that Sentient is presenting some “research on research” of this nature at the Innovation Insight Exchange in Atlanta, and I am hopeful that other researchers will also begin to focus on defining how specific measures offer unique power in explaining consumer behaviors of interest.
Looking forward to your presentation in Atlanta, Aaron!
Thank you so much Steve for this accurate and extremely useful summary of Chapter 7 of our book. I’ve been on vacation in Montana so am just now catching up. Rob and I wrote a very dense “academic” book approaching the topic with the primary mission of producing a resource for grad students and faculty wanting to apply Psychophysiology to studying media processes and effects. You have done a great job distilling our “academic” approach into a great summary for applied communication research and Neuromarketing. I’ve been blessed to now be working with two research companies, HCD Research and PlayScience, moving my academic Media Psychophysiology into the industry and am getting to see the unique challenges as well as exciting opportunities to push the research into new communication contexts with real business implications. I just want to say I feel honored and humbled to have the opportunity to even participate in this conversation and gain valuable experience. I also want to say thank you, Aaron, for your thought provoking comment and for finding use for our book. I unfortunately couldn’t make it to Atlanta but look forward to increasing my involvement in this scientific community, continuing our conversations, and even perhaps working together at some point.