How Do We Know…If Wheelbarrows Work?

by Johanna Blakley, Managing Director at the Norman Lear Center

*This is the second post in our new “Ask the Evaluators!” series, exploring real-world assessment of recent media projects to help you make informed decisions about what can be measured, and good questions to ask along the way. Stay tuned for future posts and click here to see all of the articles.*

EVALUATING “WHEELBARROWS”

Fictional feature films lend themselves very well to the characteristics of “Wheelbarrows,” usually focusing more on high-octane storytelling than pedantic social messaging. But, even though the social message might not be front and center, fictional films can be incredibly powerful purveyors of factual information, and the best of them offer nuanced treatments of complex human issues. To presume that audiences just turn their brains off while watching fictional entertainment defies decades of social science research demonstrating that media representations have a profound effect on people’s sense of social norms and even their own sense of agency. Audiences are moved by fictional representations and, if the depictions feel realistic and compelling, people often apply what they learn in fictional settings to their real lives.

So how can media researchers figure out what kind of impact a fictional film has had on viewers’ awareness, knowledge, attitudes or behavior?

While most audience members recognize that documentary films are often carefully engineered to deliver actionable data to viewers, moviegoers do not immediately assume that a fictional film might teach them something or encourage them to change their attitudes about a particular issue, or take action after the film is over. But the great advantage of trying to increase awareness and engagement in serious social issues through fictional entertainment is that it is usually far easier to attract an audience that has little to no interest in the social issue at the heart of the film. In other words, it’s a great way to reach beyond the choir.

At the Norman Lear Center, we have been evaluating the social impact of health storylines in fictional TV shows since 2001, but it was only recently that we had the chance to measure the impact of a fictional feature film, Steven Soderbergh’s science fiction medical thriller, Contagion. The film, which earned $75 million at the box office, follows the rapid progression of a highly contagious virus that kills within days.

 

As the epidemic grows, medical researchers and public health officials work to contain the disease, introduce a vaccine to halt its spread and calm the panic that spreads as fast as the virus itself. The film highlights the factors that shape the occurrence of a pandemic, the limits and consequences of public health responses and how interpersonal connections can play a role in the spread of disease. But most people did not choose to see Contagion because they had a strong existing interest in the public health issues around global pandemics. The film featured a bevy of A-list talent — Matt Damon, Gwyneth Paltrow, Kate Winslet, Marion Cotillard, Jude Law and Laurence Fishburne — which significantly increased the odds that this film would be seen by a very broad range of moviegoers, most of whom know very little about global pandemics.

WHAT WE LEARNED FROM EVALUATING CONTAGION

There is a treasure trove of academic research demonstrating the efficacy of using entertainment for educational messaging. [1] We believe our study of Contagion (a full report is available here) advances this research, particularly because of our use of a comparison group and because we were able to survey respondents years after exposure. For other media researchers who are looking to measure the impact of Wheelbarrows, here are a few pieces of advice:

Lesson 1: Hire someone who is familiar with survey research: a faulty survey instrument, or an ill-defined sample, will yield no meaningful results.

To measure changes in knowledge, attitudes and behavior we created a survey and posted it to a number of Participant Media’s web and social media sites, as well as distributing a link in their TakePart newsletter.  Fifty-one percent of survey respondents had watched Contagion and the other half had not. Given that people choose which films they will see based on their personal taste and interests, it is difficult for media researchers to determine a film’s impact without dealing with some degree of bias. In this study, propensity score matching (PSM) techniques were used to control for self-selection bias among survey respondents. PSM helped identify factors that predict the likelihood of a person seeing Contagion. The Lear Center’s research team performed a statistical analysis of survey responses from all the respondents who watched Contagion and determined what personal characteristics increased their likelihood — or propensity — to see the film. Using this data, we were able to create a matched control group of very similar people who had not seen the film.

Lesson 2: If you have the resources and the talent, create a matched control group

While a typical study of the social impact of a mainstream film might involve a national representative survey sample, which can be quite costly, we decided to deploy a survey through social networks (a convenience sample) and create a matched control group – a group of very similar people who had not seen Contagion. Among other things, our goal was to find out whether people who had seen the film were more knowledgeable about global pandemics or more likely to have taken action to prepare themselves for the kind of public health disaster depicted in the film.

*Infographic by Veronica I. Jauriqui from the report Going Viral: Measuring the Impact of “Contagion.” Courtesy of the Media Impact Project at the USC Annenberg Norman Lear Center.

Lesson 3: Take advantage of digital networks – email lists, listservs, social media networks – in order to locate a survey sample weeks, months or years after exposure.

One key problem with media impact research is that, simply because of logistics, evaluators often survey viewers immediately after they have seen the film, when the messages are still fresh and emotions may still be running high. Because we had access to social networks which included people who had seen the film, we were able to find viewers a full three years after the film was released. This allowed us to determine whether the film had a lingering effect on those who had seen it.

FINDINGS

We discovered several positive outcomes, including:

    • Increased knowledge about viruses: viewers of the film, compared to the control group, were significantly more likely to know that viruses mutate and to answer more quiz questions about viral pandemics correctly.
    • Better prepared for a viral pandemic: watching Contagion increased the odds that people prepared an emergency kit and had conversations about viruses.
    • Preventing the spread of viruses: Contagion viewers were more careful about washing their hands frequently, compared to the control group.

 


*Infographic by Veronica I. Jauriqui from the report Going Viral: Measuring the Impact of “Contagion.” Courtesy of the Media Impact Project at the USC Annenberg Norman Lear Center.

We were fortunate to be evaluating a film that was as thoroughly researched as Contagion was. One of the hazards of exploring important social issues in a fictional context is that crucial historical and scientific facts may land on the cutting room floor as creative storytellers work to craft the most compelling tale. It is a concern in documentary film, as well, but evaluators need to be particularly cautious about the accuracy of the messaging in entertainment media. Just because it feels true, doesn’t mean it is.

[1] See, for example, Singhal, Arvind, Michael J. Cody, Everett M. Rogers, and Miguel Sabido, eds. Entertainment-Education and Social Change: History, Research, and Practice. Mahwah, NJ: Lawrence Erlbaum Associates, 2004.