Earlier this week, Debra Allcock Tyler described data as often pointless, dangerous and counterproductive. Sue Holloway of Pro Bono Economics puts the case against her comments.
On Tuesday, at an event organised by think tank NPC, Debra Allcock Tyler talked about pointless, dangerous and counterproductive data collection. Those who were there would have heard my challenge to her.
How do charities know what’s working unless they actually ask people? And then you are collecting data!
I would suggest that simply relying on your personal ‘knowledge’ is also dangerous and may be counterproductive. People may smile if you have done something for them, simply to be polite, and it may actually have made no difference to them at all.
If you want to make a difference, you need something a little more objective than gut instinct. You may see a change in them, but how do you know it’s down to what you’ve done? People working in charities are no less prone to biases and misinterpretation than the ‘majority of those who use and analyse data’ who were so roundly chastised by Debra.
Suggesting that simply looking at the data would lead us to exclude old people from using the NHS is just not true. It may tell us something about how effective the NHS is in treating conditions associated with getting older, if that’s the question. And it may tell us how the cost of treating older people is changing with an ageing demographic and advances in medical science, if that’s the question we are asking of it. But data itself does not ask the question, and neither does it provide all of the answers.
Instead it sheds light on the choices we face, and the choices charities face about where they focus their effort and how they spend the money entrusted to them by others.
At Pro Bono Economics we help charities think about what data they can usefully collect in order to answer the questions around value for money that funders, and other stakeholders, may legitimately pose. We know how difficult it can be, and are clear that it should be proportionate and relevant. It may be possible to use data that others have collected to show the change you are making, as long as what you do is close enough to what others have done and tested.
Great strides are being made in getting access to big administrative datasets – which was the focus of the NPC event – which can help charities answer that all important question ‘what might have happened if we hadn’t done anything?’
We offer independent support to charities who do want to have a better understanding of the change they are making, and help to upgrade the data system in the process.
Of course data can be misused, but that should not mean that we give up altogether; we must try harder to focus efforts on getting what is useful to help answer the questions that are worth asking.
Sue Holloway, is the director of Pro Bono Economics, a UK-based charity that matches volunteer economists with charities to address questions around measurement, results, impact and value.