Eighty journalists turned up for the press preview – including almost all the top political editors – even though Tony Blair was presenting the government's Annual Report in the Commons at the same time. Normally you count yourself lucky if as many as five people show up for a screening. Many of the political journalists thought they were coming to a hanging party: they had arrived believing their own press cuttings. A common view, written up by those who had neither seen a frame of the film nor talked to me, was reflected by the ironical Guardian: “Michael Cockerell is to make a programme revealing how the Prime Minister's self-effacing official spokesman suffers daily at the hands of brutal political hacks”. I was Alastair Campbell's patsy and they were to be the scapegoats. Naturally my view was somewhat different. But I was well aware from the start of the hazards involved in trying to make a film about the state of relations between Number Ten and the media. At its simplest, how was I going to prevent the Government's spinmeister-in-chief putting the spin on us?

Over the years I had developed a wary regard for Campbell's presentational skills. One graphic example had come in the early hours of New Labour winning power on May Day 1997. Tony Blair and Alastair Campbell had flown down from Sedgefield and arrived just before daybreak at the Festival Hall, where the Labour celebrations were still in full swing. The climax would be the arrival of the victorious leader. But Campbell held Blair's car back out of sight. He was leaving nothing to chance with the grand entrance. Showing a meticulous attention to detail, he had phoned the meteorological office to find the exact moment that dawn would break and on cue he waved the Blairmobile forward. As Campbell later confided, there was no point in the Prime Minister-elect making a speech about the dawn of a new age in the dark. Campbell had learned many of the tricks of his trade as a tabloid political editor – for Today and the Daily Mirror – when he freely admits he would propagandise for the Labour party to counter what he saw as the imbalance in the press. He'd also become skilled in the techniques of radio and TV as a producers' favourite talking head.

But it had taken nearly six years to get Blair and Campbell to agree to my making a documentary with real behind-the-scenes access. I had first written to Campbell with the suggestion when he was appointed press secretary in 1994. Over the course of the next six years, as New Labour moved from opposition to government , I would repeat the request. Campbell's reply could best be summed up in the words of the pop song: “You keep me hanging on”. Sometimes he would come up to me at a party conference or in the Commons and say: “You'll get us one day” – a nicely ambiguous reply. Last October he wrote turning down my latest request, saying “it's a no, not a never”. Blair did agree to give me a long filmed interview to mark his thousandth day in office, and in February this year, he and his press secretary pushed the door ajar.

My editor Anne Tyerman and I went to see Campbell and discussed the possibility of making a film about Number Ten and the media. We said we would need real access over a period of months to both himself and the Prime Minister and to the workings of the Downing street press office; we would also want to film at the twice daily briefings, whose secrecy the lobby journalists had always jealously protected. Campbell agreed that I should sound out the political editors. Some of them were deeply suspicious of Campbell's motives, while others thought it a good idea to show what really went on. But negotiations were still going on when I had a call from Campbell. “You have really stirred it up now”, he said. I expressed surprise. “Look on the Downing Street website”, he said. There I discovered that Campbell had put out a notice saying that Number Ten had agreed to our making the film. It was the first time I had ever learned through cyberspace what I would be doing for the next few months. “Given the increasing focus in the media on the issue of so-called spin, we have agreed to cooperate with the BBC on a documentary about the way we work here in Downing Street”, announced Campbell. “I am confident that if a film explores the full range of what we do, the public will have a better understanding of the relationship between modern politics and media”.

Shadow

But my producer Alison Cahn and I weren't naïfs entering the Downing Street jungle. She was a highly experienced political filmmaker, with very good lines to the senior Whitehall mandarinate through family and other connections. I'd been making political films for 25 years and have written two books about Prime Ministers and the media. We also constantly spent our time, off camera, talking to people – inside Number Ten and outside – with firsthand knowledge of how the system operates when we were not there. And we had two further crucial weapons in our armoury to counter any dissembling. The most powerful was the cutting-room floor. Anything that we filmed that didn't have the ring of truth or we felt was put on for the cameras was consigned for horizontal filing. Alastair Campbell had no rights over the film, nor did he ask for any.

The second important weapon was the length of our access. We went on filming inside Number Ten over four months and saw things that by definition you can't see from outside the famous black door. If you are around long and unobtrusively enough, people get used to you and get on with their business. Sometimes they don't even realise you are there – as with the shots of Blair rushing down the corridor to greet a VIP who'd arrived early, or when the Russian President Putin came. Campbell himself was apparently oblivious of our camera, which caught him suggesting to a BBC journalist that he change the subject at a press conference with the Prime Minister. And there were the occasional jokey moments, when there was a cock-up with Downing Street's high technology. Also, many events we filmed were unpredictable and uncontrollable. For instance, Alastair Campbell could not know in advance of the questions that the political journalists would ask him at the lobby briefings. And sometimes he would emerge worse for wear.

It was true that Campbell didn't swear at the briefings, but he says: “If you read the coverage in the press, it would be strongly believed I spend most of my briefings f'ing and blinding and that I have stopped swearing while you're here. Well, I don't recall ever swearing in a briefing over the past three years.” He did not deny that he sometimes swore in what he called “the margins of the lobby” – coming in and going out – and there are well documented instances of the ripeness of his language to journalists on the telephone. But the worst instance of bad language in the briefings themselves the journalists could come up with was when Campbell famously – and wrongly as it turned out – denied that Blair had intervened on Rupert Murdoch's behalf with the Italian Prime Minister: “That's crap, spelled C-R-A-P.”

There was much speculation in the lobby about why Campbell had agreed to the documentary. Some supposed it was his vanity, others that it would be a hagiography, others that he was preparing some kind of exit strategy and he saw it as a good job application. I felt he and Blair had agreed to the filming – despite the objections of some political advisers and officials in the Number Ten press office – because they felt the Government could gain from an accurate representation of the state of play between Number Ten and the media. “I may be making the biggest mistake of my professional career in letting you in”, Campbell told us. “But what I want to get across is that we fulfil a basic, necessary and legitimate function. We are not the horrible, Machiavellian people as portrayed. There is a huge range of information that we have to get out in a co-ordinated way. But the press – hand-in-hand with the Conservatives – say it's all just spin. It is all part of the strategy to say you can't trust the Prime Minister; that this guy doesn't really stack up, doesn't stand for anything”.

New Labour's own private polling showed that for the public the word “spin” was taking on the same kind of negative connotations that “sleaze” had for the last Government. People who, as Paul Routledge of The Mirror put it, “a few years ago wouldn't have known a spin doctor from a spin-drier” were now deeply suspicious of the Blair project. They contrasted Labour's glossy promises on the Health Service with their own experience of waiting eight hours in casualty or eight months for cancer surgery.

But Campbell's view was that much of the public disillusion with the government resulted from the way the media covered politics. “The right-wing papers are against us in any case, and the left-wing papers are against us because we are not mad, old Labour”, he argued. “But this is a serious Government doing serious things.” Yet it was becoming increasingly difficult to get the Prime Minister's true voice across because the lobby journalists were “obsessed by trivia, splits and personalities; they treat politics as soap opera and ignore what the Government is really doing.”

Contempt

Campbell's view is that straight political reporting has become the victim of the 24 hour-a-day news media and the ever fiercer competition for viewers, readers and listeners. He claims that the old distinctions between fact and comment have disappeared along with those between the broadsheet and the tabloid agenda: “They are all in the same competing middle market. Newspapers have falling circulations, they don't have the promotional budgets they once had – so they have to do it editorially. The political journalists are all in the business of drawing attention to themselves. They try to give the sense that everything is in the fast lane – going at 150 miles an hour: everything is a scandal, everything is a controversy. I have a drawer full of cuttings going back three years each of which says: ‘It's been Tony Blair's blackest week'”.

The lobby journalists would not accept Campbell's arguments. They maintain they are essential conduits to the public of what is really happening in government. And they cannot be expected to print just what Campbell tells them – that would turn their newspapers into Pravda. They have to print stories that will interest their readers, who are increasingly turned off by politics. And they reject Campbell's view that the political journalists are the true spin doctors – imposing their own agenda and slant on what they report. “We are not spin doctors – we are professional decoders”, says Michael Prescott. “Politicians speak a strange language and it is up to us to decode it – and that leads to clashes with Campbell”.

Many of the lobby correspondents proudly produced for me personal letters they received from Campbell – tearing them off a strip. One such letter to a well-known political editor begins: “I know that questions of journalistic ethics do not keep you awake at night... The fact that you would regard ‘unprincipled and duplicitous' as a badge of honour for your brand of journalism only increases my utter contempt for you.” And Campbell ended his letter by saying that he no longer wanted to have his weekends interrupted by phone calls from the lobby man with his “half-truths, flyers, inventions and lies”. Campbell identified the system of anonymous quotes – on which so much lobby journalism relies – as the heart of the problem. He said that when he arrived in Number Ten he had brought in changes. He had agreed to be identified as “the Prime Minister's Official Spokesman (PMOS)” rather than “Downing Street” or “Sources close to the Prime Minister” – although he did not want to be named as Alastair Campbell. “One of the reasons why we went on the record as the PMOS was that I and the Prime Minister were spending so much of our time having to say to colleagues that all this stuff about a source close to him was complete nonsense. Yet still the papers are still full of: “a source”, “a friend”, “an insider says”. And in my view a very substantial selection of those quotes are simply made up.”

The Sunday Times's Michael Prescott put the lobby case: “If you want to get to the heart of what is going on in a Government you have to approach people you know and trust, and they have to know and trust you. They'll tell you what is really going on, but you've got to guarantee them anonymity because otherwise you could lose a minister his job..” It was a persuasive piece of pleading; but I wondered about source inflation. Somehow sources always seemed to be “senior” – whether backbenchers or ministers. Never was the source characterised as “a lowly researcher” or “that drunken MP who hangs around Annie's Bar and will tell a journalist anything in return for a double scotch”.

One battle-hardened veteran of the endemic conflict between Number Ten and the media paid a visit to Campbell while we were filming. Sir Christopher Mayer, now our man in Washington, was John Major's press secretary for two years as the Tory Government came under the fiercest media assault. Mayer was impressed by the way Campbell has sought to modernise the ramshackle Downing Street press machine to deal with the demands of the modern media. Campbell had set up the Orwellian-sounding Strategic Communications Unit (SCU), designed to try and dilute the lobby's monopoly on political news by making the Prime Minister available to alternative outlets, such as Daytime TV, women's magazines, regional papers and the ethnic press. There had been no SCU nor a round the clock Media Monitoring Unit in Sir Christopher's day.

Monitor

I asked Sir Christopher that he thought of Campbell's aim of seeking to be in control of the political news agenda. “Ha”, he replied, “every Press Secretary has this dream – to be proactive, to seize the agenda every morning and dominate it throughout the news cycle. And quite often with a good press machine you can do so quite a bit of the time. But inevitably, you cannot control events in the outside world. And so a lot of it is bobbing, weaving, ducking, and counterpunching. It certainly happened in my case. Alastair has experienced the same.”

What the PM and his press secretary could not have bargained for when they agreed to us making the film was that Campbell's role and the whole question of spin would become such a central part of the story. Following the presentational disaster of Blair's speech to the Women's Institute, it was revealed that Campbell would be stepping back from the front line and no longer briefing the lobby journalists twice daily. The papers reported this as Blair's symbolic defenestration of his spin doctor. In fact, as a leaked memo from the PM showed, it was a decision that had predated the WI débâcle. Blair told me: “I decided some time ago that it was sensible for Alastair Campbell – just because of the sheer pressure of work, apart from anything else – to step back from doing all the lobby briefings every day. Otherwise he can't also look at how we are trying to get the Government's message across, or what we are saying in a broader sense – because he is just sitting there doing his equivalent of Prime Minister's Questions every day.”

Campbell said: “We had – and I particularly – had got myself into a situation where combat was the only language being spoken, which is not necessarily terribly sensible either way.” He said that he was now going to concentrate on strategy to ensure that the Government's long-term message and achievements across the board were effectively put over. This was not he insisted anything to do with election planning, although he conceded it could have beneficial side-effects for New Labour. But he added: “Although you can't be fighting over every headline, we will still have proactive, aggressive communications strategies; because if you don't – with the modern media the way it is – then you are not going to get your message across and you are not doing your job properly”.

Yet the decision to restrict Campbell to little more than one lobby briefing a week, brought cries of outrage from the political editors. Many of those who had launched the most baleful attacks on Campbell and the Government as evil geniuses of spin queued up to appear on our film to demand his return to the lobby fray. They claimed: “Alastair is the authentic voice of the Prime Minister – not just policy but Blair's personal reactions. We can't do our job properly without him”.

As Tony Blair put it in a part of his interview we did not use: “Some of the journalists complain to me now and say: why isn't Alastair Campbell doing the lobby briefings any more? And I say , well for goodness sake: you complain when he does them and you complain when he doesn't. But we can't spend our entire life worrying about what these guys are thinking otherwise I wouldn't do my job as Prime Minister”.

After our documentary went out, there was much comment about how the Lobby seemed to be suffering severe withdrawal symptoms: it had long been living in an abusive relationship with PMOS, but now it couldn't bear to be parted.. The editor of The Guardian, Alan Rusbridger, claimed that the two sides were “trapped in a downward spiral of passive-aggressive behaviour”; something had to be done “to untie the knot before Westminster politics and press become terminally polluted by mutual cynicism and disrespect”. His solution was to end the semi-anonymity of the lobby briefings by naming Campbell and his deputy – and to adopt a far stricter code on anonymous pejorative quotes by politicians. “We will encourage reporters to be as specific as possible about the source of any anonymous quotation”, wrote Rusbridger.

The political editor of the Daily Express, Anthony Bevins, mocked Rusbridger's plan: “If the use of The Guardian as Gordon Brown's and Peter Mandelson's personal website is going to be exposed, I shall be impressed”. Bevins claimed that what emerged from the film was what he called “the insidious laziness of the hacks: the lobby system is a crutch for crippled journalism”. And he advocated opening up lobby briefings to all-comers: “They should also be televised and the Prime Minister should hold regular on the record press conferences.”

Nearly 20 years ago, Peter Hennessy, David Walker and I had made similar proposals in our book Sources Close to the Prime Minister. In it, we had argued that the pervasive secrecy of Whitehall was matched by a largely pliant lobby, content to kiss the chains that bound it. A good deal has changed since then, under successive Governments: there is far more official information available now than ever and many of the old secret taboos of Whitehall have been lifted – on such subjects as the Budget, the existence of security services and the rules governing the conduct of Cabinet ministers. And Alastair Campbell himself has taken steps to end the mystique which old lobby hands liked to preserve, first by his semi-emergence from the closet as PMOS, then earlier this year by putting summaries of his lobby briefings – excluding the political exchanges – on the Number Ten website. It was all a very far cry from the lobby's own rulebook which said: “Never talk about lobby briefings either before or after they have happened – especially not in the presence of those not entitled to attend them.”

Perhaps most important of all, Campbell and the lobby did agree to our camera filming at the hitherto secret briefings in both Number Ten and at the lobby's own room in the Commons. But when he gave his consent, Campbell was quick to point out that the decision was not a precursor to televised lobby briefings: “Our view is that Downing Street briefings should remain on the record, but off-camera. Ministers will continue at all times to present the Government's case to Parliament and the public.” But it seemed his experience of the film might have made him more amenable to the idea of the TV cameras going in on a more regular basis. His general view of the programme was that, on balance, it had justified the risk he and the Prime Minister had taken in giving us access. “I would give it seven out of 10”, he said. There was now more public understanding of the job the Downing Street press office does – but he felt that we had failed to portray the journalists as “corrosively cynical” enough. We had not dwelt sufficiently for his taste on the regular diet of “trivia, froth, speculation, and sheer invention” that passed for so much political journalism these days. Why had we not shown, for instance, how many re-shuffle stories were just plain wrong? So many ministers regularly forecast for a move or the chop were still in place – and the media never acknowledged their mistakes. “But when I get one thing wrong, it's all over the papers for days”.

I was pondering his strictures when the phone rang. It was one of Campbell's special bêtes noires from the Sunday press. What, he wanted to know, was the reaction in Number Ten to the Rory Bremner depiction of the Blair-Campbell relationship? I said that I had not talked to anyone there about it. He then presented his own thesis that Blair must have found it all highly damaging – did I accept this? I said I did not know, but suggested he draw his own conclusions. The next day, his thesis was presented as fact and to support it he wrote: “A Downing street insider said: ‘you must draw your own conclusions'”. I've been called many things, but never that before.