NAO Data Report: Gov't Spending Up to 80% on Cleaning, Merging Data

The National Audit Office (NAO) has released a report today following its investigation into how the government and Cabinet Office use data: it is not a happy read, with highlights including the fact that in some departments, 80 percent of time is spent cleaning and merging data.

Here are three key takeaways from the NAO data report.

1: Data is Not Seen as Important

In its investigation the NAO found that within nearly all government departments data is not seen as a priority. In fact the only department that refers to data as a “strategic asset” in its department plan is the Department for Digital, Culture, Media & Sport.

The NAO found that: “Five of the seven large customer-facing departments have their own data strategies, but these vary in their maturity. For example, in one department the strategy has been approved but no additional funding has been agreed to implement it. In some departments data strategies are still under development, or elements of data are incorporated into other strategies.”

The NAO notes that the government has encountered numerous issue arising from bad data management yet: “Despite years of effort and many well-documented failures, government has lacked clear and sustained strategic leadership on data. This has led to departments’ under-prioritising their own efforts to manage and improve data.”

2: Not Joined-Up

The NAO notes that several incidents in recent years have highlighted the need to obtain and maintain good quality data. For example the Windrush “situation” demonstrated how bad decisions can be made off the back of poor quality data; in this case the ability to correctly identify individual’s rights.

The NAO also cited the Carer’s Allowance, which was allocated new systems: but these systems did not have all of the underlying data required, such as length of employment or allowable expenses; this resulted in officials having to input data manually.

In its report it found “more than 20 ways of identifying individuals and businesses across 10 departments and agencies, with no standard format for recording data such as name, address and date of birth.”

“The problem is replicated in local areas where information is recorded differently across local and constitutional boundaries. This makes it difficult for government to maximise its data asset, for example by allowing thematic analysis across different sectors to help understand economic challenges or systemic problems.”

“For example, where names vary across systems, where National Insurance numbers may not match or where data is recorded differently in differing local systems. In our discussions departments have suggested that between 60% and 80% of time is spent cleaning and merging data. In some areas this can equate to several hundred analysts’ time.”

3: “Clearly Articulated Plan Needed”

The NAO has cited the need for the Cabinet Office to include a ‘clearly articulated’ plan of work to address the issues around data safeguarding and department methods to use data in more efficient manners.

It noted that: “Often departments have produced a new customer interface, for example rolling out the online application for the state pension, without fundamentally reconsidering what data their department needs and whether the current data remains fit-for-purpose.”

The NAO’s three key recommendations for the Cabinet Office is that they need to put in place governance for data, set out data requirements in business cases and train front-line staff in the correct methods of handling data.

Commented on the its report Gareth Davies, the head of the NAO stated that: “Government has lacked clear and sustained strategic leadership on data, and individual departments have not made enough effort to manage and improve the data they hold. This can reduce public confidence in government’s ability to collect and use people’s data effectively.”

“The right processes, systems and conditions must now be put in place, otherwise the new data strategy will become yet another missed opportunity.”

The Cabinet Office has been contacted for comment.