The Hard Work of Measuring Social Impact

Quantifying performance and measuring results are no longer the sole domain of for-profit enterprises.
Today, many nonprofit organizations also find themselves on the hot seat—not with stockholders but with donors who expect similar levels of accountability to show how their money was spent and what that spending achieved. Yet there has been little agreement on a set of hard-and-fast metrics to measure social performance. How should a nonprofit manager respond when a significant donor asks for proof of his or her contribution's impact on a particular social issue?

"There are two big conversations among nonprofit leaders," says HBS associate professor Alnoor Ebrahim. "One is around accountability. The second focuses on performance, particularly impact."
Ebrahim is teaching in Performance Measurement for Effective Management of Nonprofit Organizations in June, a joint executive education program presented by the HBS Social Enterprise Initiative, Harvard's Hauser Center for Nonprofit Organizations, and the Harvard Kennedy School. He will co-chair the HBS Executive Education program Governing for Nonprofit Excellence (with professor Herman "Dutch" Leonard), for nonprofit board members, in November.

In two working papers that break down what makes social impact easier or more difficult to measure (one coauthored with HBS professor V. Kasturi Rangan), Ebrahim offers nonprofit managers a clearer sense of how to respond to accountability demands from a variety of constituents. (These papers are: "The Many Faces of Nonprofit Accountability" and "The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance".

Prioritize For Accountability
For starters, Ebrahim counsels against the default position that nonprofit organizations are accountable primarily to funders, simply because that's the money source.

"Nonprofit leaders must prioritize among accountability demands—from foundations, government agencies, individual donors, and their own staff and clients—otherwise they'll be pulled in all directions," he says. "They cannot be accountable to everyone for everything. Rather than aiming simply to comply with the demands of the most powerful actors, nonprofit leaders need to focus their attention on accountabilities that really matter for achieving their missions."

Assessing an organization's impact on a large-scale societal issue such as poverty is a complex and costly effort. In the aftermath of Haiti's earthquake, for example, an organization like Oxfam America could be expected to provide data on short-term results, such as the number of people sheltered and fed. But it would be much more difficult to measure how Oxfam's emergency relief efforts play a role in its overarching, long-term mission of finding lasting solutions to poverty and injustice. Because there are so many players involved in the relief effort—governments, other nonprofits, and business—it's difficult to isolate the influence of any given organization.

That isn't to say that measuring large societal change should never be attempted.

"It can be done, but it requires a longer time horizon and an effort to understand the contributions of many organizations working in the same place at the same time," Ebrahim notes. "Delivering emergency relief and basic services in sanitation, water, and housing is easier to measure than impacts on public policy or on good governance, freedoms, and rights. Societal transformations—such as improving human rights or democratic conditions—involve multiple actors and causal mechanisms that are still poorly understood. In such cases, it can still be useful to try to measure what an organization is doing and whether its strategies are working to influence societal change. But there's a tradeoff in terms of the best potential use of scarce resources."

Not For Everyone
To help nonprofit leaders determine when they should reasonably attempt to assess impact, Ebrahim offers a contingency framework for measuring performance.

"The basic idea is that not everyone needs to measure impact, and that what you should measure is contingent on what you're trying to achieve," he states. "That's affected by two things: your theory of change and your operational strategy."

A theory of change is an organization's rationale for how its actions will ultimately lead to the achievement of its goals and mission. "How well understood is the relationship between cause and effect?" asks Ebrahim. "Some organizations have a focused theory of change, where the relationship between their actions and their impacts on society are linear and can be observed." The delivery of a basic service, such as shelter for homeless people, can be understood this way, Ebrahim explains. You can count the number of people being fed, sheltered, and clothed.

"The implication for managers is to figure out where their organization sits in the contingency framework."

"A more complex theory of change would be that in addition to sheltering people, getting them out of poverty requires greater access to education, health care, and employment resources; at the same time, you might be trying to influence public policy on issues that affect the homeless. That involves a whole set of interrelated activities that are broad in scope. As you widen your scope to deal with a major social problem, the harder it becomes to measure your impact because it is tougher to isolate cause and effect. It's no longer a simple linear relationship, but a complex set of relationships."

A nonprofit's operational strategy is the other important aspect to consider, he adds. An organization with a focused strategy concentrates its resources on a specific task or intervention—such as the homeless shelter that focuses on providing overnight beds and a warm meal, or an ambulance service that focuses on getting patients to the emergency room quickly.

But humanitarian relief and development organizations like Doctors Without Borders, responding to a disaster, will necessarily have more complex operational strategies. In order to provide shelter and food, they have to control an elaborate supply chain, from procuring tents, food, water, and medicine, to airlifting those supplies and distributing them on the ground with trucks and staff, all the while coordinating with other NGOs, governments, UN agencies, and the military.

Despite all that, the organization alone has limited influence over what happens in the country as a whole, simply because there are so many players involved, and the tasks so interwoven. "Measuring one organization's contribution to earthquake relief may be possible, but it is less important than collective results," Ebrahim says.

On the other hand, nonprofits that are the only actors working in a region exercise a higher degree of control. As an example, Ebrahim cites his research on two NGOs in western India: the Sadguru Water and Development Foundation and the Aga Khan Rural Support Programme. These nonprofits work in hundreds, if not thousands, of rural communities on issues relating to water provision, irrigation, forestry, and energy.

"They're major players by virtue of their scale and the scope of their activities, and in some communities they have been the only consistent service providers in a generation," he observes. "They can assess their performance much further down the logic chain due to their relatively high degree of control and the length of their involvement in the same communities."

A Matter of Resources
So, if an organization has a focused, or linear, theory of change and a tightly focused operational strategy, measuring results is much easier. But these are likely to be short-term outputs rather than longer-term societal impacts.

What about with a more complex operational strategy? "As long as your theory of change is linear and clearly understood, you can begin to get at outcomes and impacts," says Ebrahim, "because you have a sense for what connects cause and effect." This applies to the delivery of basic services such as health care, immunization, education, and employment programs.

"Highly intelligent, thoughtful people end up developing some very different approaches to measuring social performance."

In an environment where the theory of change is less well understood, such as policy- and rights-based work, where collaboration may matter more than acting alone, an organization's resources would be better used pursuing its mission and developing approximate measures to help it improve, rather than chasing hard proof of impact. Many international NGOs like Oxfam, Save the Children, CARE, and ActionAid face this challenge. They have complex theories of change, complex operational strategies, and limited control over their environments. Ebrahim suggests that "it is more important for them to have learning systems in place, for adapting to complex contexts, than it is for them to seek conclusive proof of impact."

"The implication for managers is to figure out where their organization sits in the contingency framework," says Ebrahim. "They may find that donors want them to measure long-term impact, but they're in a situation where their focus should be on short-term results. In those instances, they're caught between a rock and a hard place. They want to demonstrate accountability to funders, but at the same time they need to push back and share their analysis to explain why they're not measuring impact."

There's risk to this approach, of course—but Ebrahim points out there's also potential for real gain in building trusting, transparent relationships with donors.

Accountability isn't a one-way street, either. The downward pressure on nonprofits to demonstrate results and impact puts the onus on donors as well.

"Donors want to support nonprofits that can demonstrate impact," Ebrahim says. "But donors have a role to play in enabling impact, and they have an opportunity to connect nonprofits that are working on different pieces of a social problem in isolation. This might mean building the capacity of those nonprofits to have an impact at scale together.

"If a foundation says it'll only fund organizations that can demonstrate they do good work to improve educational outcomes for at-risk youth, it sounds reasonable enough," he continues. "But if the foundation picks only those organizations that are able to show results, it may end up leaving critical gaps in the chain of changes that need to occur to achieve long-term results. We often talk about how a nonprofit organization should have a theory of change, yet many donors don't have one of their own."

Internal Benchmarks Possible
Despite the ambiguity surrounding social impact metrics, Ebrahim says nonprofits can nonetheless establish their own internal benchmarks to assess performance and determine whether they are achieving their mission.

"In these complex challenges, especially policy and advocacy, it's important to develop a clear strategy that lays out how you expect to influence particular actors," he says. "And obviously there should be a method for periodically testing and critiquing that strategy, and modifying it if it's not working."

Ebrahim is working on a series of case studies focused on nonprofit and public organizations that are considered innovators in the area of measuring social performance, such as the Acumen Fund, the Robin Hood Foundation, the Millennium Challenge Corporation, and ActionAid International. ActionAid, for example, is practicing "downward accountability" by creating transparent measurement processes in coordination with the communities it serves so that those communities can offer frank feedback on the organization's work.

Ebrahim's research suggests that measuring results is valuable as a discipline for nonprofit managers, even if they don't find universal metrics for social impact.

"It turns out that highly intelligent, thoughtful people end up developing some very different approaches to measuring social performance," he says. "That suggests we might not see agreement on a common set of metrics in the social sector for a while. What we could see instead is convergence on what constitutes critical reflective learning processes within the organization."

Julia Hanna is associate editor of the HBS Alumni Bulletin

This article was provided with permission from Harvard Business School Working Knowledge.