I received a letter last week from a very senior civil servant warning me not to reveal in this column any "privileged information" acquired as a minister. But I have a revelation of another kind to make. Readers would be shocked not by what ministers know when they are taking decisions, but by what they do not know, and are not told.
When it comes to the secret world of GCHQ and the US National Security Agency (NSA), the depth of my "privileged information" has been dwarfed by the information provided by Edward Snowden to the Guardian. The cabinet was told nothing about GCHQ's Tempora or the NSA's Prism, or about their extraordinary capability to vacuum up and store personal emails, voice contact, social networking activity and even internet searches.
I was also on the National Security Council, attended by ministers and the heads of the secret and security services, GCHQ and the military. If anyone should have been briefed on Prism and Tempora, it should have been the NSC. I do not know whether the prime minister or the foreign secretary (who has oversight of GCHQ) were briefed, but the NSC was not.
This lack of information, and therefore accountability, is a warning that the supervision of our intelligence services needs as much updating as their bugging techniques. The state should not feel itself entitled to know, see and memorise everything that the private citizen communicates. We cannot even rely on incompetence as a bulwark for our freedoms. The state increasingly has the capability to retain everything as the cost of computer memory collapses.
I have been shocked but also mystified by Snowden's revelations. Throughout my time in parliament, the Home Office was trying to persuade politicians to invest in "upgrading" Britain's capability to recover data showing who is emailing and phoning whom. Yet this seems to be exactly what GCHQ was already doing. Was the Home Office trying to mislead?
The latest name for its super-surveillance project is the communications data bill, touted by the home secretary, Theresa May, and stalled by Nick Clegg. It comes with a £1.8bn price tag, so this is no small decision either politically or financially.
At present, police, tax staff and others can ask communications providers for records of the contact between a suspect and other people. They are not allowed to intercept the content of the calls without a warrant signed by a secretary of state, but they can analyse the listing of numbers and the duration of calls – as you get on your phone bill – to search for gangs, drug dealers or terrorists.
The weak Blair-Blunkett era safeguards on these trawls need only the approval of a police inspector, and there are more than half a million a year. The sign-off should be tougher. After all, information that someone is calling Alcoholics Anonymous or the Samaritans may be embarrassing and damaging, although neither illegal nor suspicious. Information is power, and the necessary corollary is that privacy is freedom. If GCHQ cannot keep its own secrets, what price keeping yours or mine?
Of course, investigators find this who-calls-whom data useful, and want the communications data bill because more calls are being made outside the traditional telecoms providers, often through non-UK outfits such as Viber and Skype. The Home Office argued that we need a new law to make these providers retain records – including emails and phone calls – to provide to the police and MI5.
The civil liberties implications are nauseating, but my inner economist also doubts this government IT project could come in on budget and specification. How long would it take the bad guys to switch to services in the 15% of all contacts not covered by the programme? Would we spend all that money merely to keep tabs on the lawful and the stupid? But the question is not the risk-reward balance of the Home Office programme, but why it needs the communications data bill at all. GCHQ and the NSA are tapping into the fibre-optic superhighways, collecting and storing not just the who-calls-whom data but the communications themselves.
Some of the explanations for this political mystery are not pretty. Maybe GCHQ is not as confident as it claims about the legality of what it has been doing. Maybe it has even been jurisdiction-swapping (though it denies this). Under this long-suspected arrangement, GCHQ would bug people that the NSA wants to bug, but is not allowed to do so because they are Americans. And the NSA would return the favour. Both could then protest their lawfulness.
There are other explanations. Maybe GCHQ is not as capable as its now-leaked boasts claim: every public body has a budget to protect. Or it might have been embarrassing, pre-Snowden, to admit to GCHQ's capability, so maybe the securicrats thought that £1.8bn was a modest price to duplicate what they were already doing. I doubt, though, that hard-pressed taxpayers would agree.
Whatever the explanation, the Home Office was happy to allow the NSC and the cabinet – along with parliament – to remain in utter ignorance of Prism/Tempora while deciding on the communications data bill. The Snowden revelations put a giant question mark into the middle of our surveillance state. It is time our elected representatives insisted on some answers before destroying the values we should protect.
