You Know Your Film Has Made a Difference When…

If you were sitting on the shuttle bus next to me in Park City, Utah at Sundance Film Festival two years ago  and you weren’t too dressed up, I may have handed you this card.

If you had turned it over, here’s what you’d have read:

:

Highly unscientific research, you might say? Perhaps. But, as the expectations for “films that change the world” heightened, I wanted to get some first-hand feedback about the way “impact” was being considered within the creative community.

Below fleece beanies, some eyes rolled. A few down-enhanced shoulders shrugged my questions off. Others knew exactly what I was talking about and didn’t hold back. These socially-committed storytellers rattled off key accomplishments or moving anecdotes that they felt proved how their films had made a difference. From “Strangers came up to me after screenings saying I’d just changed their lives” to “My film helped turn the tide on legislation,” there was no doubt that many filmmakers have specific indicators for effecting change.

The postcards led to focus groups with Tribeca Film Institute, International Documentary Association and Active Voice to answer this question: when it comes to creativity, what can we really measure, and how?

Of course, practioners and scholars have been researching the impact of various films and programs for decades, but over the last two years the question of measurement has heated up. For example, I was surprised to see the rise of the “self-evaluating media maker” (there’ll be more on that in later posts; sign up for our blog.)

A range of new innovative toolkits and platforms underscored the burgeoning approaches to assessment, such as Sparkwi.se (way back in 2012), BRITDOC’s Impact Evaluation Plan, the iSchool at the University of Illinois Urbana-Champaign’s ConText and Harmony Institute’s StoryPilot. There are more players and greater diversity of opinion: some creatives are excited about the possibilities of being able to deepen their strategies through real-time assessment, while other voice concern about choosing the right method, the mounting pressure for metrics, and the threats to creativity. Many, like me, believe that all of these trends are in play.

In a variety of settings – from those early shuttle chats to our PDQ (“please don’t quote”) sessions and, more recently – in workshops, three themes have really stuck with me:

• Most media makers wished funders could understand how their relationships with and knowledge of their subjects might contribute to outcomes different from what the funder expected, but significant in their own right. (Don’t miss Fledging Fund’s perspective.)
• Media makers had dramatically different opinions – and varied rationales – about whether they should be the ones to gather and analyze data for evaluation.
• Almost every media maker I’ve talked to expressed interest in communicating about measuring impact without threatening their chances for funding.

Creatives aren’t the only ones who have a lot to say and profound questions about measuring the impact of story-driven content. Funders do too, especially those who see creative media as a powerful way to advance the particular change their foundation wants to see. Frequently, if they’re going to make the case for supporting media makers, they need to know what’s working and why. Professional evaluators are often expected to produce impressive data that demonstrates change, even when the outcomes may be hyper local, or subtle, or incremental.  Let’s also not forget that the people who count on media to help engage hearts and minds – advocates, activists, teachers and more – also want to know more about how and what kind of stories contribute to social change.

What started as a postcard, with three lines for comments or a 15 minute conversation is now www.HowDoWeKnow.net, currently in beta. My objective – through the website, workshops and other exchanges – is to create a space where these often sensitive and far-reaching issues can be discussed, and where knowledge can grow (you’ll find bushel of garden metaphors at Horticulture – and you can add your own here.)

Take a look and let me know what you think!