The Best Big Data Projects Start Small
Much of the hype around big data carries an image of large companies using esoteric hardware and software to run incredibly complex analytics. But the reality is that many big data projects are actually quite small, use straightforward technology and achieve real world aims.
Today, thanks to the cloud, it is more than possible to run smaller scale projects with minimal up-front investment. Local authorities are already showing interesting results in areas from early intervention and social care to traffic management and recycling and rubbish disposal.
Get the Fundamentals Right First
It is crucial to get the basic infrastructure in place before embarking on a big data project.
It is about getting the fundamentals right, not building a massive data centre and hiring lots of people in white coats to run it.
The key barrier for most organisations is improving on existing infrastructure to allow better access to the data that is already being collected. Local authorities are no different to most organisations in finding that data is often stuck in siloes within departments, or even individual business units, and is almost impossible to access and use. Breaking open these siloes requires cultural as well as technological change. But cloud technology and software-based networking can make this process far easier than it was in the past.
Investing in open source, cloud-based infrastructure can help create systems where data can be accessed by applications across the organisation or by partners outside, as long as there is cultural support too.
Once data is available, it needs to be checked to ensure it is clean and fit for purpose. This is another vital step which can often be overlooked. Most big data projects need to spend a sizeable fraction of their budgets on data processing and cleansing before any analytics is even attempted. This process allows you to get the very best out of data you own, only then do you need to consider if you need to import data from elsewhere.
Content from our partners
Once you have the infrastructure and clean data sets in place then the real work of analysis can begin. Again cloud technology can help here with lots of cloud apps available to help with early stage analysis.
Although the implications of moving to a more data-driven organisation can be profound some of the most successful examples start with quite modest goals.
Identifying a single issue and using data to solve it can help prove the usefulness of big data strategies to the whole organisation and win over doubters. Data science is a complex and specialised business and most organisations find it easier to work with a partner with relevant skills.
It is also crucial to ensure the right data governance and privacy controls are in place from the very start of any project. Again ensuring your fundamental infrastructure is sound and solid will make compliance with GDPR and other regulations much easier.
Start Simple
Northamptonshire County Council took such a step-by-step approach in order to reduce congestion and make spending on transport more effective. The council hired the University of Northampton to collate data on journeys made by students, health care workers, patients and council staff. In total this provided information on journeys made by 32,000 people, a sizeable proportion of total travel in the county.
Quite simple analysis found that almost half of dedicated patient transport to hospital could be provided in other ways. Analysis of other council transport contracts found spare capacity of almost 1000 places which could provide more cost effective travel for some of these patients.
Over and above this, the project, oneTRANSPORT, is also helping to inform future decision making around expanding the university to ensure that transport and congestion implications are properly considered and planned for. The council is now partnering with other local authorities and technology providers to deepen the understanding available.
Building on the success of such projects allows local government to take a more innovative approach to how services are delivered. There are big opportunities not just for savings but also for radical improvements in services in areas like social care and early intervention policies.
Next Steps
Innovation charity Nesta’s discussion paper “Datavores of Local Government” looked at some emerging trends in how local councils are using their data to provide savings and improved services. One important shift is a move to predictive analytics. Moving spending from solving problems when they occur to providing them earlier has long been a goal for local authorities. Historically this has been stymied because the financial burden can often shift to another organisation and it has been difficult to accurately measure the success of such early intervention.
Predictive analysis can help remove these barriers.
In the US a nationwide NGO is using predictive analysis to identify which expectant mothers would most benefit from visits from specialist nurses during pregnancy and the first two years of the child’s life. By targeting resources and testing the best time for intervening the charity is able to fine tune how its resources are spent and improve outcomes for both mother and child. This sort of predictive analytics can offer radical solutions by allowing authorities to intervene before rather than after problems arise.’
Big Data is not Going Away
There is no way for local authorities to ignore the ever increasing wave of big data.
As more services shift online, more communication with residents is digital, more council staff use mobile devices and more infrastructure is linked via the Internet of Things – so the scale of the data lake will continue to grow.
Getting the right infrastructure in place means rather than councils just coping with this increase in data, they can take real advantage and find genuine value in the insights it can provide.
In the next few years how local councils deliver services, and even the types of services they offer, will be profoundly changed by the impact of data analysis and also machine learning.
Embracing this will enable services to continually evolve and improve as citizens’ needs change.
