Thanks to Ed Snowden, the whole world now knows about the NSA’s surveillance capabilities. In his thoughtful and timely piece on The Strategist, Klee Aiken asks if the mass surveillance of big data really keeps us safe.

Data mining technologies now allow our intelligence and security agencies to collect unprecedented amounts of personal information. Quoting a former US intelligence official, Klee describes the current approach as not so much looking for a needle in a haystack, rather ‘collecting the whole haystack.’

There’s probably no way to prove whether this is an effective approach: as Klee observes, this is all hidden behind ‘layers of security clearance’. Coincidentally, the day that Klee’s post was published, the US National Journal magazine ran a long article by Michael Hirsh, which examines the NSA’s very wide data-trawling methods in the context of the changing threat from al-Qaeda.

Hirsh argues that AQ is moving from the 9/11 spectacular type attacks on iconic targets to more lone wolf opportunistic strikes, like the Tsarnaev brothers at the Boston marathon. Hirsh describes how Abu Musab al-Suri (a nom de guerre that means ‘the Syrian’) has emerged as an increasingly influential jihadist leader, and how al-Suri has long argued for lower-level attacks.

In his post, Klee states that bulk data gathering won’t always identify and thwart the ‘lone wolves’, with lower levels of organisation and competence. But Hirsh argues that the NSA’s seemingly indiscriminate scouring of phone data and emails is ‘precisely what intelligence officials say they need to detect the kinds of plots al-Suri favors’.

Hirsh cites Michael Hayden, former NSA Director, as saying these kinds of attacks will be harder to track and that the US should expect more, less lethal attacks than in the past. Hirsh also quotes Mike Rogers, chairman of the US House Intelligence Committee pointing out that AQ’s capabilities for a strike in the US are more dangerous and numerous than before 9/11. Also cited is a US government official 'well versed in NSA practices' who says that trying to tie specific intelligence 'tidbits' to specific foiled plots is too simplistic:

That entirely misses the point. It doesn't account for the reality of how intelligence works. It's not that pods or cells are disrupted by one piece of information from one authority. It's a complex endeavor that puts different pieces together to rule things out.

Hirsh’s answer to Klee’s question on the effectiveness of big data is that if we’re to find the needle of al-Suri style plots, then we’ll probably need the ‘whole haystack’ approach to phone and email data for some time: ‘for better or worse, the only hope to track them all is an exceptionally deep, organized, and free-ranging intelligence apparatus’.

Klee's post concluded by pointing to big data's cost to privacy. If Hirsh’s judgement is right about the need for a big data approach, then we’re going to need to devote much greater efforts to developing a coherent privacy doctrine that addresses the challenges of our enhanced ability to collect, analyse and distribute personal information.

Anthony Bergin is the deputy director at ASPI.