Dialsmith

Image courtesy of BBC

Just listened to this recent podcast from The Market Research Event Blog featuring Netflix’s Director of Consumer Insights Dave Decelle in which Decelle explains why his Consumer Insights team has foregone traditional recall-based research methods for those that are aimed at capturing “Moments of Truth.” Here at Dialsmith, we talk about the concept of uncovering “Moments of Truth” as a unique benefit from using our tools and services, so we were very interested in what Decelle had to say.

As background, Decelle’s Consumer Insights team interacts directly with Netflix customers to try to understand their behaviors and decision-making processes. In this podcast, he shares the methodologies they use to capture what he (and what we here at Dialsmith) call “Moments of Truth.” These are the key moments where customers are actively experiencing or making purchase decisions. The challenge for researchers is that these moments are fleeting and cannot be captured reliably after the fact due to memory bias, recall issues, and interference from outside sources. So, innovative research pros, like Decelle, are searching for effective in-the-moment methods to tap into the minds of consumers. Here’s how Decelle explains it in his podcast:

“We felt that just putting people in a focus group or a survey and asking them to recall, ‘Hey, the last time you sat down to watch something, what was happening around you? What was going through your head?’ People are terrible at recall. So, instead we took this very in-the-moment approach to the moments-in-truth research.”

To test the effectiveness of their methods, Decelle’s team ran an A/B test using a traditional survey that collected feedback after the fact and a survey that asks participants questions via a mobile app as they were watching something. Comparing the results, Decelle’s team noted that the in-the-moment data was richer and revealed much more about the experience than feedback from the traditional survey. As Decelle notes, “We found that there was definitely a difference in the data in terms of which drivers really mattered. We chalked that up to the fact that recall was really poor.”

Dialsmith’s research clients have expressed similar challenges with studies that rely on recall-based feedback alone, which is why they have interest in developing studies that integrate our in-person or online dial testing tools.

In-the-moment dial testing results helped NYU’s Dr. Charlton McIlwain zero in on moments of truth in race-based political ads.

A good example of how in-the-moment methods can change the dynamic of a study is the work done by New York University professor Dr. Charlton McIlwain. Dr. McIlwain, whose been studying the impact of race-based messaging in political ads for more than 10 years, introduced Dialsmith’s Perception Analyzer dials in his most recent testing. Unlike previous studies, Dr. McIlwain was able to use the dial data to pinpoint specific moments of the ads that drove the strongest reactions from participants and make the correlation between those “Moments of Truth” and the introduction of both implicit and explicit race-based messaging. Per Dr. McIlwain:

“The dial results clearly showed that there are a lot of complicated and sophisticated things going on over the course of an ad that have an impact on where people end up at the end. We were able to see the movement (of the dial result lines) at precise moments—the clear static lines and then the stark point where the ratings start to fall in very close proximity to the moment when the race-based content was introduced.

Our primary finding was that people reacted most strongly at the moment where race-based content was introduced in both the implicit and explicit conditions. We also had a surprising finding: participants reacted more negatively to the implicit condition than the explicit condition.”

As both Decelle and Dr. McIlwain have concluded, in-the-moment methods are proving to be an integral component to understanding today’s consumer and viewer behavior. Here at Dialsmith, we are excited to be a part of this growing area of research and are interested in hearing your take on it or any experiences you’ve had. We invite your comments and feedback below.