Author Christopher Andrew on MI5’s Secrets

For much of its 100-year history, spanning two world wars, the Cold War and the fight against terrorism, Britain’s MI5 domestic intelligence agency has operated in total secrecy. The government acknowledged the existence of the service only in 1989 and publicly identified its leaders in 1992. Now, as part of efforts to make its operations more transparent, MI5 has given unprecedented access to its files to Britain’s foremost intelligence scholar, Christopher Andrew, whose new book, The Defense of the Realm, is considered the most complete history of the agency ever published. TIME spoke with Andrew about the conspiracy theories he’s debunked, former spies in the British government and his feelings about James Bond.

Were you given full access to MI5’s files up to the present day?
I was given virtually unrestricted access. However, I did not ask for access to 21st century files of cases that haven’t yet come to court or that are still under investigation.

(Read “British Spies: Licensed to Be Gay.”)

What was the biggest surprise you uncovered?
One discovery made a real impression on me. The highest calling of any intelligence agency is to tell truth to power. The first example of this I came across is during the 1938 Munich crisis, one of the most ignoble moments in British foreign policy. MI5 at that point actually understands [Adolf] Hitler far better than the British government and then Prime Minister Neville Chamberlain, in particular. He pays no attention to what they tell him during the negotiations that lead to the Munich agreements. So Vernon Kell [the head of MI5 at the time] decides the only way he can get through to the Prime Minister is to tell him what Hitler is saying about him, specifically that he calls him as an arsehole. He gives him the German word that Hitler is said to use: Arschloch.

(See pictures of Hitler’s rise to power.)

Do you think that transparency helps intelligence agencies in the long run?
Yes. The idea used to be that you don’t want the public to know anything, so you don’t tell them anything. What changed a generation ago is that the British people became less deferential, and if they’re not given some idea of what’s going on, they fall for conspiracy theorists. The best-selling book in the U.S. about British intelligence is, after all, Peter Wright’s Spycatcher. A couple of the stories that he put in there that are complete nonsense are still widely believed: that there was a plot against former Prime Minister Harold Wilson — I now know there wasn’t, though there was a file kept on him — and that the head of MI5 for nine years, Sir Roger Hollis, was actually a Soviet agent. The files disprove this too. If you don’t have information, you get disinformation.

Looking through 100 years of files, what do you see as MI5’s biggest success?
Definitely against Germany during the Second World War. Some of the best German spies were turned into double agents, and that is the central part of the most successful deception in the history of warfare, without which the D-day landing couldn’t have taken place. What I discovered in the files is how it all began — in a characteristically eccentric British way. The most adventurous of the MI5 agents in the 1930s was an air ace from the First World War named Christopher Draper. He’s called the “Mad Major,” because he was absolutely obsessed with flying under London’s bridges. He’s invited over to Germany. Hitler is very interested and spends over half an hour talking to him at an air show. When he gets back [to London], he’s asked to become a spy for German intelligence, and he says fine, stopping only to ask MI5 if that’s O.K. on the way! By monitoring the contacts he has to make with German intelligence, MI5 picks up a German agent they didn’t know about and begin to roll up the entire German network.

So MI5 has had its share of characters?
So far as I know, MI5 is the first government agency to actually identify a sense of humor as a desirable trait in a recruit. When you look at the world of espionage, it’s the only profession in which a fictional character is at least a hundred times better known than any real character. James Bond is the only brand leader to have remained in the top spot over the years. An intelligence officer once told me that in the depths of the Third World he met a tribal leader whom he thought knew no English but who addressed him with the famous phrase “Hello, Mr. Bond.”

(See pictures of James Bond.)

What about MI5’s biggest failures?
Those are related to the biggest failures of Western intelligence generally. When MI5 begins to get involved in Ireland in the early 1970s — at that point they knew less about Belfast than they did about Nairobi — well, I haven’t come across a single file that relates intelligence during the Troubles that begin in 1969 to intelligence between the Easter rising in 1916 and the founding of an [Irish] Free State in 1922. Files from that previous period show that intelligence was incredibly confused, and poorly coordinated with local police. What happens in the 1970s? Exactly the same thing. It’s a nearly perfect example of the old adage that those who do not understand past mistakes are doomed to repeat them.

Taking a long-term view might also have helped better predict Islamist terrorism in the 21st century. MI5 was slow to realize the potential threat, as it was busy dealing with IRA campaigns in the 1990s. It’s not until 2003 that MI5 grasps that the radicalization of minority British Muslims is leading to terrorist attacks on British targets.

(See pictures of terrorism in the U.K.)

You discovered that three Labour Members of Parliament under Wilson were considered Soviet agents. Did that come as a shock to you?
It did come as a bit of a surprise. What surprised me most, however, was the degree to which getting too excited by the threat of communist subversion was not usually done by MI5 but rather by government. Labour leadership sent MI5 a list of names of 16 Labour MPs who they thought were more communist than Labour. MI5 refused to get involved because it saw this as party politics. However, the man at the top of that list [Will Owen] we now know had been a Czech agent for years!

Read “British Spies: Licensed to be Gay.”

See TIME’s Pictures of the Week.