New technology has made big data analysis possible, but John Tate warns that its success still depends on having good people and good processes in place.

I'll start my first column of the year with some good news.

Some 93 per cent of all accountants working in the charity sector are feeling positive about the year, and 97 per cent were keen to get back to work after the December break.

Some 73 per cent predict an increase in income for their charity and 84.7 per cent expect Arsenal to win the Champions League in 2014. At that point, unfortunately, I then woke up.

Big data is another topic everyone expects to produce good news this year. It promises to bring together information from a wide range of sources to analyse, interpret and even predict future trends across all parts of our personal and working lives.

So how close are we to being able to confidently produce this sort of information, and is it time for charities to invest in the ‘latest and greatest’ big data offerings?

To get the benefit of any investment in IT, three things need to come together: people, processes and technology.

Technology

Looking first at the technology; big data solutions have actually been around a long time, but they have just been known by different names.

So, for example, back at the start of my career – some 35 years ago – I worked on an MIS project (management information system) for part of a FTSE top-20 company.

This project was to collate data provided to us on the spending patterns of UK consumers. The system summarised millions of individual purchases and produced a wide range of reports on them. We then used forecasting tools to predict future sales and market share.

Later I worked for a fashion business where we implemented EIS technology (executive information systems), designing a system to track sales of products, by outlet, by day.

From a technology perspective, a great deal has changed in recent years. Hardware costs have fallen hugely, and the internet has spawned a vast amount of online data – from social media to e-commerce transactions.

So charities now potentially have access to a great deal more information on their donors, partners and beneficiaries. Software products make it ever-easier to combine data from different sources and present it in ever-prettier formats.

The good news is that technology solutions that might have cost hundreds of thousands of pounds to implement 25 years ago can be bought for only hundreds of pounds today.

People and processes

However, from a people-and-process perspective things have not advanced so quickly. The unfortunate reality is that our business processes are often inefficient. You need good business processes to get good-quality data, and people are not very good at managing the sort of change required to improve business processes or implement a new piece of technology.

I mentioned a couple of the information projects I have been involved in. The first, with the FTSE company, failed completely. The second, with the fashion retail business, was a great success and implemented before the mainstream adoption of the internet.

All items to be sold were tagged at head office with a small punchcard which held details of the product and sales price. A weekly list of all goods coming into head office was supplied to the third-party ‘big data’ supplier.

When the stock was dispatched to the shops, a second list was produced. Then when the shops sold an item the tag was removed.

Once a week, after close of business on Saturday, all these tags were collected by the supplier. They were read and processed by a computer on Sunday and a comprehensive set of merchandising reports was delivered back to head office for 8am on Monday morning.

So complex processes can be implemented and made to work – with or without technology – and big data projects can be successful.

However, please remember that however whizzy a piece of technology, people and processes need to be very carefully managed if a big-data project is to have any chance of success.

John Tate is a business consultant, IT adviser to CFG and a visiting lecturer at Cass Business School.