What are we to do with Big Data? Edward Snowden has kick-started a public debate about the legitimate scope of intelligence in a world of digital interconnectivity, and last week Klee Aiken asked us to consider the effectiveness of intrusive surveillance programs which analyse bulk information. There's a robust debate to be had about the nature of modern privacy. But precisely because it's so controversial on legal grounds, Aiken maintains that the indiscriminate collection of telephone and internet data by the NSA needs to prove its value as we ponder what to reform. Along with some of the US politicians who oversee the system, he's pessimistic that the game is worth the candle.

In response, Anthony Bergin drew attention to the shifting operational template of terrorism. Echoing a point made by IISS’s Nigel Inkster , Bergin suggested that metadata techniques will be needed to trace the development of home-grown radicals, who have become more of a threat in recent years. The pattern of future attacks are likely to be dispersed and irregular, receiving inspiration from al Qaeda through the very channels that prove most susceptible to signals intelligence.

Both made thoughtful points—and neither believes that much can be adequately settled without more detail. But I fear that Aiken and Bergin, despite their differences, have given short shrift to Big Data surveillance by unnecessarily narrowing the terms of the debate. Internet and telephony metadata techniques were designed to intercept the communications of terrorists, and they've largely been defended on those grounds. But the debate about metadata analysis shouldn’t be limited by how much we know about terrorist plots being intercepted. If we're to judge how privacy protections should constrain intelligence, it's worth considering the full spectrum of threats that can be transmitted via modern telecommunications, and which Australia and its Five Eyes allies are capable of monitoring.

This means, above all, that no discussion should take place without due consideration for the future of cyber security. Unfortunately, few commentators have explored this angle of the story, but the impact here could well be the most lasting consequence of Snowden’s revelations. For instance, beyond the embarrassment and backlash from allied leaders, an immediate casualty of the NSA leaks was the US campaign to pressure Beijing over its widely-suspected cyber espionage. In the early months of this year, the Obama administration was carefully laying the ground for a diplomatic outreach to European leaders, who it hoped would help to shift the calculus of China’s leaders. Post Snowden, however, with its own dirty tricks exposed, the US has lost all diplomatic leverage on the matter.

Similarly, more's at stake in the debate about metadata and privacy than our ability to ward off physical terrorist threats. The kind of intrusive electronic surveillance methods exposed by Snowden are likely to offer the most effective basis for defending telecommunications networks against cyber infiltration, as inconvenient as this is for our concern with privacy. This was highlighted in a September address by James Baker (PDF), a former Justice Department official who represented the Bush administration before the Foreign Intelligence Surveillance Court. According to Baker, surveillance of internet activity will be necessary to detect all kinds of malware in the future, and the large-scale retention of data in the corporate world can make sure that a forensic investigation is carried out after a particularly sophisticated attack.

In other words, an ambitious counter-intelligence effort is probably the most effective way to guard against the transmission of malware across computer networks. Indeed, the NSA had plans to build a system that monitored all Internet packaged data entering the US, and was piloting a system which protected the American high-tech defence industry. Again, post-Snowden, this appears to be dead in the water.

I doubt the Australia Signals Directorate has the inclination or the clout to press for this kind of system right now. But, as with most things cyber-related, it's difficult to forecast where we'll be in a few years. We don't know the exact cost to our national prosperity from organised crime or foreign espionage, and we can’t predict the severity of many cyber weapons. With the growth and diversification of malicious cyber activity, who knows what the public will want after the Snowden leaks begin to dry up.

This doesn't resolve the debate about electronic privacy. Instead, it complicates our choice. Perhaps, if we don't intend to examine the content of electronic transmissions, as Baker calls for in the US, we might still empower the Attorney General to receive metadata on the flow of internet traffic. This would establish a baseline of normal cyber activity which should help to detect serious anomalies. But whatever we decide to do, our understanding of electronic privacy will need to adjust to the reality that Big Data has a valuable role to play. The infrastructure of modern surveillance, arguably wasted on the hunt for terrorists in haystacks, might be needed after all.

David Schaefer is a sessional tutor in the School of Global, Urban and Social Studies at RMIT University.