First, a word definition. The term lobby correspondent is itself confusing. No newspaper any longer calls any of its political journalists a lobby correspondent. Moving to Westminster for the Financial Times in 1981, I successfully pressed for the by-line of political editor rather than lobby editor, which I regarded as symbolically important. Lobby correspondents are, in practice, the main political news reporters of a newspaper, as they are now formally described. Their distinguishing feature is that they cover politics from their offices in the press gallery at Westminster, covering the Prime Minister, the Government, the Opposition, the political parties and Parliament. The term, "the lobby", refers to the list of accredited correspondents kept by the Serjeant at Arms who are allowed access to the members' lobby and some adjoining areas.

Much nonsense has also been written about the Downing Street briefings. A common charge is that lobby journalism is inherently flawed because of the rules of unattributable briefings – and the resulting close relationship created between press officers and journalists. (I hate the term "spin doctors" not only because they lack the professionalism of doctors or the subtlety of spin bowlers but also because of the false implication attempting to influence the media was somehow an invention of the mid-1990s). When I first joined the lobby in 1981, there was still something of quasi-masonic secret society with references to red and blue mantle (briefings then by the labour of the Opposition and the Leader of the Commons). The Lobby was dominated by a small group of mainly middle-aged men who liked to sustain the mystery, partly to enhance their standing and their independence of their offices.

Lady Thatcher

The sole significant difference from what happens in the White House is that the briefings are not held on camera, as Nicholas Jones of the BBC has been advocating in a series of books. Mr Campbell and his predecessors have argued that televised briefings would be incompatible with primary accountability to Parliament. The differences with Washington are anyway much exaggerated. From my own experience there 10 years ago, there was a full spectrum of briefings – from public to deep background. Secretaries of State, such as Henry Kissinger and James Baker, often moved in the course of a single briefing from being cited by name to being a senior State Department official to the even vaguer senior administration official. The real difference with Washington is that officials are instinctively more willing to answer telephone queries from reporters and provide information.

Collective briefings , like those conducted twice a day when Parliament is sitting, have now become a minor source for any semi-competent political journalist. They tend, however to be confused with the close relations, and unattributable conversations, which lobby journalists often have with politicians and their advisers. Some who want to "abolish the lobby", whatever that means, want to get rid of these contacts as well as the mass briefings. But all specialist journalists have unattributable conversations with their sources. In many ways, the links between some other specialist groups – in, say, sport or motoring – are far cosier with their sources than at Westminster where the very numbers involved prevents too close a relationship from developing. Or, rather, if one journalist becomes too close to a certain politician, then his or her rivals, both in journalism and politics, will ensure that an alternative viewpoint is reported. To insist that all contacts between journalists and politicians should be public and attributable is daft and unworkable. Of course, there are risks involved in unattributable contact, both of cosiness and distortion, but that depends, as always on the quality of the journalist involved.

No golden age

Jack Straw, who organised a study of the decline in the mid-1990s, identified a number of explanations: the televising of Parliament from 1989; a delayed reaction to the large parliamentary majorities of the 1980s (though coverage remained low during the Major years when votes were less predictable); a generational change among political editors who were less interested in traditional gallery coverage. He commented that: "It seems to be essential that newspapers should once again start seriously to report what happens in the chamber and to show how, through debate and argument, the most serious conflicts are resolved through politics". But there is a non-sequitur here. Newspapers and broadcasters certainly have a responsibility to report on politics, which many fail to fulfil. But that is not the same as reporting what is said by MPs in the chamber. Debates and exchanges on the floor of the Commons are now only a small, and increasingly marginal part of the political scene. MPs are confusing symptoms for causes. The real issue is that Parliament itself is increasingly being bypassed.

Compared with the 1950s, and even perhaps the early 1970s, the floor of the House of Commons is no longer the central arena of politics. The sources of political news have changed out of all recognition. Even within Westminster, there is now the expanded select committee system as a generator of stories. A multiplicity of broadcast outlets exists from dawn to midnight, all seeking the views of politicians. Most policy announcements are foreshadowed by pre-briefing by ministers and their advisers – and often an appearance by a minister on the Today programme when he or she pompously says they cannot discuss the details of the White Paper or whatever (why then appear on the programme?) and then promptly does so. The later statement in the Commons is, the March Budget apart, usually an anti-climax, thinly attended and thinly reported. This is partly a circular process since ministers know the best way to secure coverage on their own terms is by pre-briefing, rather than waiting to reveal the full details on the floor of the Commons, then that is obviously what they will do.None of this is a sudden creation of the arrival of Alastair Campbell in Downing Street. The specific Blairite innovation has been using specially staged events, like appearing at a hospital or school to provide a favourable, and non-adversarial backdrop to an announcement. These changes in themselves do not threaten the position of lobby correspondents, as opposed to the now largely extinct species of gallery reporters and parliamentary correspondents. Indeed, the growth of pre-briefing of announcements before they are made on the floor of the Commons has enhanced the role of senior political journalists as the conduits of such hints and leaks. There are dangers of creating a dependency relationship whereby a minister or briefer only gives details to a favoured journalist – and the journalist feels compelled to play along because his or her news desk will be critical if the paper fails to get the story first. The real challenge to the position of lobby correspondents comes from the shift of power away from Westminster – to the broadcasting studios, to the courts, to European institutions, to the new devolved assemblies in Scotland and Wales, to new semi-independent regulators (setting everything from interest rates to electricity prices), and to the courts (where judges decide highly political issues). The Blair Government's programme of constitutional change has increased the number and influence of such alternative power centres, while the centrality of Westminster has been further reduced by the very size of the Labour majority in the Commons and the irrelevance of most votes and by Mr Blair's own apparent indifference to much of what is said there.

Uncomfortable

Equally important has been a change in the nature of the media itself. Increased competition, both among newspapers and broadcasting outlets, has resulted in a significant change of approach. The price war and increased competition among the broadsheets has led to a shift down market on at least the news and feature pages of many papers. The emphasis is on attracting marginal and younger readers who are believed to be less interested in reading about "heavy" politics – that is policy rather than personality. So while many broadsheets, if not tabloids, still carry plenty of space devoted to politics, the style has changed. An increasing number of stories are about scandal and misconduct rather than policy or procedure. Splits and disputes are exaggerated. The outspoken maverick, always described as a senior backbencher, can be sure of being quoted no matter how insignificant he is. Nuances are lost. Big occasions or reports are still covered extensively, but there is seldom any consistent follow-up. Political reporters often despair of getting any stories into their papers about policy discussions, or the work of think tanks. I have read little of any significance out of Westminster on, say, public service agreements which are crucial to the Blair Government's delivery of its promises on improving standards in public services.

Media commentators and academics, such as Bob Franklin of Sheffield University, have noted a more critical tone, "more polarized, less measured and less willing to be neutral in its appraisal of parliamentary affairs". There has also been a generational change among political journalists. Many of the younger,ambitious correspondents have seen that the route to attract notice, and promotion, is by getting scandal and split stories. Any sense of proportion, let alone reflection, is often lost in the eagerness to produce what they believe their news desks want. That has been reinforced by one of the inherent weaknesses of the lobby system – the fact that all the journalists work together in the cramped surroundings of the Press Gallery. That reinforces a herd approach – both the adoption of a common view of a story and the related defensive "protect my back" attitude. This produces a sameness of approach rather than diversity. Journalists wait around late at night to see the first editions of other papers and are then asked by their night desks to match them. That occurs even more on political stories than on others because the journalists from rival organisations work together.

Political journalists themselves have not been directly guilty of the greater intrusion into the private lives of politicians that has developed during the 1990s. That has mainly been done by general or investigative reporters. But political reporters know that a damaging story about a politician is a quick route to the front page. This shift is reflected in the spread of the "laddish" culture to the press gallery, despite the increased number of female political journalists. Remarkably few seem interested in the latest political books or biographies, except where they provide a good story. This is not just a British phenomenon. Similar complaints have been expressed in Washington about an increased focus on scandal, rivalry and conflict to the exclusion of policy and legislative process. As too often in Britain, politicians are portrayed in an unfavourable light, on the make. Political journalism in Washington has often appeared to be a succession of scandals with the White House press corps collectively pursuing allegations out of all proportion to their significance.

Decline

Reporting of politics is also being changed by the Internet. That offers the potential to broaden public access to information which newspapers no longer report. For instance, Hansard and other parliamentary papers are now available on the Internet, resulting in much greater public awareness of what is said on the floor of the Commons than was true a few years ago after the broadsheets cut their gallery coverage. David Mckie recently wrote a report on "Media Coverage of Parliament" for the Hansard Society which looked at the decline in traditional gallery reports. He concluded on an optimistic note because of the increased use of the Internet. "There is simply no point in lecturing newspaper editors on their dereliction of public duty in abandoning Westminster coverage. Those days are not going to come back; they don't even need to. If people are eager to know what the Commons and Lords are saying and doing, they'll have all the access they need. Whether Parliament can succeed in commanding the people's attention then is, of course, a quite different matter". Lobby, and political journalism is in a period of rapid change, more than most of its practioners realise. Yet the role for political journalists close to those exercising power will remain. Not all those in power will necessarily be in Westminster or Whitehall. But we still need sharp and assiduous political journalists – whose role was defined by Shakespeare in King Lear – "hear poor rogues talk of court news; and we'll talk with them too, who loses, and who wins; who's in, who's out".