Roy Head and Jennifer Steel argue that the finance function should be at the heart of impact evaluation.
The civil society world is a funny creature. Despite much talk about measuring impact, the sector’s structure makes this very hard.
In the private sector it’s simple: companies rise and fall based on profitability. Executives win promotions not only for creating profit, but also for closing down loss-making units. In science it’s a similar story: Nobel prizes are given to those who disprove theories, as well as those who prove them.
Measurement, rigour and ruthlessness are at the heart of both capitalism and science, two of the great edifices of modern society.
The charity sector is 'opaque'
One reason why the not-for-profit sector might seem a poor relation of these two giants is that it’s almost as if it was designed to be opaque.
Imagine our organisation gets a grant from the Department for International Development (DfID).
At the end of the grant period we have an incentive to report how well it went; our future grants depend on it. And DfID managers have every incentive to agree; they want to tell their superiors that they’ve spent taxpayers’ money wisely.
There’s no natural tension or rigour in this process; no market; no science. It’s small wonder that our sector is populated with well-meaning but inefficient ideas and projects which never get closed down.
Our organisation, Development Media International, couldn’t get away with this. We make entertaining radio and television campaigns to persuade people in developing countries to adopt healthy behaviours (breastfeeding, hand-washing, etc).
The public health sphere is dominated by the sciences of epidemiology and health economics. Simply put, every health intervention is evaluated in terms of how many lives (or disabilities) it saves, and what each extra year of healthy life costs. So we must prove to funders that our approach works.
They asked: “How many lives can you save? At what cost? How can we compare you to vaccinations, or nurses?” And so we spent a year working with two world-renowned professors from the London School of Hygiene and Tropical Medicine. We constructed a mathematical model to predict how many lives we could save, and at what cost.
The predictions were exciting. Firstly, that we could reduce child mortality by around 15 per cent per annum, and secondly that the costs per life-year saved were lower than any existing intervention.
For example, primary health care for mothers and new borns costs $82 to $409 per life-year saved. HIV/Aids treatment costs much more: $673 to $1,494. The cheapest intervention ever found is childhood immunisation which costs $8 in Africa and $16 in Asia. Our campaigns are predicted to cost between $2 and $9.
If proven, this would make mass-media campaigns the cheapest way of saving children’s lives that currently exists.
So what? This is all very nice for our organisation, but can this really be applied to the rest of the voluntary sector? Aren’t some things just too difficult to measure?
Need to cooperate
It’s certainly a challenge, but perhaps we can take a lead from epidemiology, while avoiding its academic complexity.
All that is needed, in each field, is a commonly-agreed unit of impact, something that represents an outcome (eg the cost of making one child literate) rather than inputs (more teachers).
It would take real cooperation within each sector to agree on a set of units which could be comparable. But, once agreed, the finance professionals could take over. One of their special skills is reducing complexity to simple sets of numbers.
If finance staff in charities were included as key members of monitoring and evaluation teams, they could also measure their organisation’s performance in terms of unit costs, using commonlyagreed professional standards.
Agreeing those units will be a major challenge: it’s taken epidemiology many years. But as a sector we have everything to gain from being accountable and rigorous. And we already have the finance staff to lead the way.
Roy Head is CEO, and Jennifer Steel is finance manager, of Development Media International
