*This is the third post in our new “Ask the Evaluators!” series, exploring real-world assessment of recent media projects to help you make informed decisions about what can be measured, and good questions to ask along the way. Stay tuned for future posts and click here to see all of the articles.*
Activists, filmmakers, and philanthropic supporters of both often claim that social issue documentary films can be social change catalysts. Documentaries and other kinds of films have the potential to put new issues on the public agenda and to change the framing already existing challenges. But not all social issue films succeed in putting new issues out in the ether, nor do all filmmakers have this as a goal for their films. An obvious question is under what circumstances can a documentary film contribute to new or changed public discourse and social change? Doc Society (formerly known as BRITDOC) and a team of academic researchers each independently tried to answer this question in two different case studies.
GASLAND
According to Active Voice Lab’s Horticulture framework, a shovel is as a documentary film that is, “usually investigative in nature, digs for the truth, and exposes alarming information.” This definition implies that successful shovels contribute new information into the public domain, and that the alarming nature of this information will have an effect on audiences.
Josh Fox directed the documentary film Gasland, a Shovel. With this film, he had the explicit goals of injecting new information into the public sphere and generating a broad debate about hydraulic fracturing, or fracking, and its associated health and environmental effects. The film and its engagement and distribution strategy were designed with this goal in mind.
Released in 2010, Gasland follows Fox as he crisscrosses the United States, meeting individuals who have been personally affected by fracking. The film is perhaps most remembered for introducing scenes of flaming tap water, polluted by the chemicals used in fracking. The film was screened at multiple film festivals, including Sundance. The film’s team designed a campaign to screened Gasland in communities across the US, of which Fox personally attended approximately 180.
In the months and years after the film was released, “fracking” became a commonplace word, the image of flaming tap water iconic, and governments across the country instituted a variety of policies about fracking, the most well known being the Marcellus Shale Moratorium in California. But what is the relationship between Gasland and the social and political changes?
DOC SOCIETY (formerly BRITDOC)
Doc Society’s Gasland case study, featured in the organization’s Impact Field Guide, offers one example of an effort to document Gasland‘s impact. Doc Society’s valuable Impact Field Guide has resources geared toward filmmakers navigating the impact space, especially in the context of the contemporary funding atmosphere. It’s worth noting that Doc Society’s impact resources assumes the user is interested in contributing to social change.
Doc Society produces impact reports that are case studies of individual documentary films, which have all been finalists of the Impact Award. A case study considers the elements of a project, as well as the broader social cultural, and political context to determine what factors likely contributed to a change or phenomenon. At the time of writing the Gasland case study, Doc Society used a standard template that included the following:
In the case of Gasland, Doc Society’s impact report states: “The goal of Gasland’s engagement campaign was to place the 
film as a focal point in gaining support for the worldwide fight against fracking.” The description of the campaign includes description of the “extensive screening programme” organized by the Gasland team and attended by director Josh Fox. Screenings were held in communities across the country and for specific stakeholder and power holder audiences.
The report asserts that:
“All of the outreach campaign’s achievements – the legislative victories in NY and Pennsylvania, the media attention, the support of government officials – would not be possible without rise of a truly populist pro-sustainability movement that arose because of Gasland.”
HARMONY INSTITUTE & VASI ET AL.
Harmony Institute also analyzed Gasland‘s impact with a research project conducted by University of Iowa sociologist Bogdan Vas and other researchers (see article “‘No Fracking Way!’ Documentary Film, Discursive Opportunity, and Local Opposition against Hydraulic Fracturing in the United States, 2010 to 2013” for their findings). Vasi et al employ standard sociological research methods such as content analysis to show how Gasland’s screenings contributed to media coverage of fracking, social network conversation around the issue, social mobilization, and, ultimately, local policy change. Content analysis is a method analyzing text in order to identify patterns. It is particularly useful to analyze texts to determine when new phrases or ideas are introduced into public discourse and to follow how they spread through networks to determine the impact of a shovel, as a shovel is often introducing new information into the public sphere. A researcher can then compare content before and after a film airs to determine its influence over time.
The researchers measure Gasland’s influence on public attention using web and YouTube searches for both Gasland and fracking; Gasland’s influence on Twitter chatter through analysis of tweets that mention Gasland or fracking; Gasland’s influence on mass media coverage of fracking through newspaper articles that mention Gasland and fracking; Gasland’s influence on discussion about fracking (including ban and moratorium) both on Twitter and in mass media; and the effects of Gasland screenings on anti-fracking mobilizations and municipal bans in the Marcellus region.
Vasi et al find that Gasland had significant influence on public attention to fracking, especially on Twitter. They also argue that by focusing on flaming water, the film significantly influenced the discussion about fracking by turning it into an environmental and water issue, or creating a “discursive opportunity.” Finally, they analyze exogenous variables – external factors not related to Gasland or screenings of the documentary – and find that success of social movement and municipal bans on fracking are correlated with population size, income, and NGO density.
Strengths & Weaknesses
DIY vs. Employ an Expert
Doc Society’s template provides a framework that a filmmaker can use to conduct impact reporting that makes use of information filmmakers have easy access to (e.g., budgets, audience size, campaign description, awards, high-level impact the filmmaker is aware of). In an environment where impact measurement is often perceived as an additional and burdensome task for filmmakers, and can be resource intensive, this approachable reporting mechanism’s appeal for filmmakers is obvious.
However, this narrative approach to impact reporting can lack rigor or generalizability. The impact associated with Gasland – individuals signing petitions, social movement activity, and political response – are tied to the film through assertions of causality without an actual research methodology being applied. So while the links are undoubtedly there, the strength of the relationship between Gasland and social mobilization against fracking, which ultimately resulted in policy change, remains unknown. It is important to note that Doc Society’s impact reports are aimed primarily at filmmakers wishing to organize their own campaigns and are intended as learning tools.
In “No Fracking Way!” Vasi et al clearly identify multiple characteristics that may contribute to an outcome (independent variables) in communities where Gasland was screened, such as population size, relative proximity to oil wells, and strength of civil society (measured by the presence of NGOs). By measuring each of these characteristics and correlating them with success of social movement activity and policy change, they provide insights about conditions that increase the likeliness of social mobilization after social issue documentary screenings. Future filmmakers and impact producers could use this information in developing their own campaign strategies.
While “No Fracking Way!” has generalizable and shareable takeaways that could provide valuable information to filmmakers and impact producers designing engagement strategies for their films, peer review journal articles are not written with a lay-audience in mind. Harmony Institute published a brief summary on their website.
Timeline
Neither impact analysis looks deeply at the time period leading up to Gasland’s release and screening campaign. Doc Society states that, “Before Gasland arrived, fracking was a non-issue.” Vasi et al include only five months prior to Gasland’s release in their analysis, while they include 35 months after the film’s release in the impact analysis. It’s probably true that “fracking” was not a significant issue in the public sphere – mass media or social networks – prior to Gasland, supplying evidence for this claim would strengthen these analyses.
Bias
Doc Society’s methodology is to solicit impact from filmmakers, campaign impact producers, and partners who used the campaign in their work. Filmmakers have incredibly valuable information about their film’s engagement campaign strategies and impact. For example, the report includes responses from screening participants who communicated their change in perception about fracking and indicated their intent to take action. However, relying heavily on a single perspective results in a kind of myopia; important external factors that contributed to social change in the same issue area may go unrecognized or underappreciated. Currently, many impact campaigns lack the budget for more substantive, mixed method evaluations.
Vasi, et al rely on multiple data sources to show the impact of Gasland, avoiding the pitfalls of relying on one perspective. However, they rely heavily on quantitative research methods that can miss important texture, details, and facts that are important in social change processes. For example, the analysis they do linking screenings to subsequent social movement activity in communities assumes that screening participants included activists and others who were swayed to become activists through the screening experience, and especially the scenes with flaming faucet water. But because no surveys, interviews, or focus groups were conducted with screening participants, the researchers cannot be sure if: a) this assertion is true; or b) which elements of the film were most effective in converting interested parties into activists.
CONCLUSION
Gasland undoubtedly played a significant role in increased public attention to, informed debate about, and official response to fracking in the US. The two impact case studies discussed above help us to better understand the circumstances that made this possible.
If you want to better understand the impact of a shovel, both a case study or a research project that includes content analysis and a comparative analysis of unrelated, external factors can be useful.
If you’re going it alone:
If you’re partnering for impact analysis:
CONTACT US
© 2013-2023 ACTIVE VOICE LAB
Active Voice Lab is a project of Community Initiatives
Website by Computer Courage. Maintained by RED.
Lead video © Simon Christen.