It took more than 25 years for Duncan Campbell to finally publish confirmation of the Echelon project, completing a story he began breaking in 1988.

The scoop, released on The Register and The Intercept this week, capped off some 40 years of investigative journalism on British and American spy agencies, Campbell having begun his career by revealing the existence of Government Communications Headquarters (GCHQ).

By Campbell’s own phrase Echelon was "the first ever automated global mass surveillance system", appearing decades before Edward Snowden leaked documents proving that GCHQ and its American sister the NSA had set up programmes with similar ends.

In an interview with CBR, Campbell lays out how spying has changed – and not changed – throughout his career, as well as the latest attempts by politicians and spooks to justify the mass invasion of privacy, and what it bodes for the future of big data, privacy and security.

‘A passion for civil liberties’

At the height of his fame (or perhaps infamy), Campbell faced prosecution under the Official Secrets Act for allegedly receiving classified information.

This followed his arrest in February of 1977 with fellow journalist Crispin Aubrey after a meeting with John Berry, who had previous worked in signals intelligence for the British government.

In the end, all three defendants in the ABC Trial (so-named after their initials) walked free, with Berry given a suspended jail sentence and both journalists sent away with a conditional discharge, the judge ruling that the pair should not be punished for breaking a law that would be repealed a decade later.

But what motivated Campbell to get into the sort of work that can end you up in court?

"The underpinning which emerges is a passion for civil liberties and just government, and that goes together with what goes on in secret," the journalist told CBR.

Born in Glasgow and later educated in Oxford in physics, Campbell left education as a trained scientist, but started writing for the New Scientist and an alternative paper called the Brighton Voice in the 1970s.

"There were many clues as to the tapping of the undersea cables in the 1970s," he said, explaining how he became interest in government snooping. "Clues emerged from the Watergate hearings and some of the investigations that were done in the background."

After the ABC Trial in 1977, Campbell went on to track down the location of a phone tapping centre in the UK, whilst also delving into the cost of British spying to the country’s taxpayers. In 1987 he produced a documentary on a £500m satellite scheme called Zircon, a snooping scheme that was eventually canned.

These stories eventually led to Echelon, which can be seen as a forerunner to the programmes that Snowden would end up revealing in 2013. The automated computer system was capable of intercepting communications through satellite, phone, telex, telegraph, computer, and even the early Internet.

However when Campbell detailed this in 1988 the story was largely ignored. Later a European Parliament investigation in 2000-01 led to demands that something be done against it. However, a week after the recommendations were passed, two planes were crashed into the World Trade Center in New York, shelving any proposed action.

"If it’s there, they want it."

Fast forward some 12 years to June 2013 and Snowden was revealing confidential documents to the journalist Glenn Greenwald (who later founded The Intercept) and filmmaker Laura Poitras.

At this point Campbell’s work had taken him away from spying into Big Tobacco and tax havens. But what did he make of the Snowden leaks?

"I think most of us were not surprised at the technology that was out there," he said. "All those years down the line it’s incredibly comprehensive how you can build a surveillance system for the Internet."

He noted that in some ways Echelon was little different to Silicon Valley tools that people now use every day such as Google Alerts, a means of monitoring noteworthy content online. The only exception is that the NSA tapped private rather than public communications.

As Campbell puts it, spying is subject to its own version of Moore’s Law, the idea that chip processing power would double every 18 months. "If it’s there, they want it," Campbell said, referencing spies thirst for ever more data on citizens.

Yet even having said that, the journalist saw encouraging signs that British spies had a more nuanced response to Snowden’s leaks when he attended a conference in May organised by The Ditchley Foundation, a think tank that specialises in Anglo-American relations.

"It was quite an eye-opener to see that intelligence chiefs had a much more reasoned idea of what they did than the politicians," he said. Remarks made at the event cannot be attributed, but among the spies present was Robert Hannigan, director of GCHQ.

"Big data Scoops"

As far as public attitudes go to privacy (and by implication, security) in a post-Snowden world, Campbell remains pessimistic.

"My broad take on it is that it’s not looking very good," he said, noting that the French have just pushed through their own version of the US Patriot Act, which authorised many of the schemes Snowden uncovered.

Even though the Official Secrets Act under which Campbell was prosecuted has been repealed in part, journalists handling spy stories can still face pressure from government, the Guardian having been made to destroy computer equipment involved in the Snowden leaks.

Does this mean things are much the same as when Campbell started his career?

"What is different in the age of the Internet is the amount of information you can get is huge," he said. "We live in the age of big data and that includes big data scoops."