They are frustrating, however, because their sheer bulk makes thorough searching of them practically impossible. In theory the newspapers are all there to be read – not just the national daily and Sunday papers, but local and weekly titles from every corner of the country going right back into the 18th century and including many that survived only for a few issues. They are all held in the enormous redbrick hangar at Colindale in North London, which constitutes the newspaper section of the British Library. It is very easy – at least if one lives near London – to imagine that one can just nip up there and quickly find what one is looking for. In practice, one is rapidly overwhelmed by the mountain of densely-packed newsprint. Local papers are the most manageable, because they tend to run to relatively few pages weekly while being clear and repetitive in their layout. If you are trying to trace the activities of a prominent individual, such as the local MP – as I have done in exploring the early careers of F E Smith (in the Birkenhead News in the 1880s), Nye Bevan (in the South Wales Weekly Argus in the 1920s) and Margaret Thatcher (in the Finchley Press in the 1950s) – you will find that the local papers used to report their speeches, fete openings and hospital visits in full and respectful detail. Even today that is probably still true, at least compared with the scant and trivialising coverage of politics in the national press.

The national papers are much harder to search – and less fun too, partly because for understandable reasons of space and conservation they are all now microfilmed. Instead of turning the pages of a bulky volume, as one used to be able to do, allowing oneself to be diverted by the cricket scores and the advertisements (very good for period atmosphere), one has to wrestle with a recalcitrant machine and whiz through each monthly reel at headacheinducing speed – which is hopeless unless you know exactly what you are looking for and on which day. There are an awful lot of days in every month, and modern papers have scores of pages, often in several sections. One frequently sighs for the days of wartime paper rationing, when each paper was limited to about eight pages.

Another problem is that each title is microfilmed separately, so that in order to see how the press in general covered a particular event one has to send for and then plough through each paper in turn. One of the most timesaving aides for the historian would be a microfilm (or better still a volume) in which all the principal titles were filmed (or bound) together day by day, so that one could read the coverage of, say, the assassination of J F Kennedy or the resignation of Harold Wilson simply by calling up the papers for November 23 1963 or March 17 1976. But that, sadly, is a pipe dream.

Finding a major story like that is easy, however. The real problem arises when you don’t know exactly what you are looking for. You think there may be – there must be – a needle in the haystack, but you don’t know where to begin to look for it. This is where an index is vital. The trouble is that until recently very few newspapers kept indexes. The Financial Times has had one since 1981, The Guardian since 1986. The Independent missed the chance to keep one when it launched in 1986, but started one in 1992. Among the major provincial papers, The Glasgow Herald (as it was then known) kept an index from 1907 to 1974. Some local papers – such as the Stirling Journal and Advertiser – maintained one for several decades in the 19th century, while the local history group in Hull has heroically compiled an index to the Hull Advertiser and Hull Times from 1794 to 1945. But none of the popular national papers has ever had one, which is a huge gap.

Only The Times has had an index almost from the beginning. For the first century or so – 1790-1928 – there is Palmer’s Index, a freelance effort and highly eccentric, but better than nothing. Then in 1906 The Times itself – recognising its pre-eminent status as the national paper of record – started an official and comprehensive index, which continues to this day. Moreover, in 1978 it still took its responsibility to history seriously enough to commission an additional index for the first five years, 1785-90 (when it was still the Daily Universal Register), which Palmer had not covered.

This priceless tool is the reason why historians rely so heavily on The Times. It is not because the so-called “Thunderer” is necessarily the best or most authoritative paper at any given period – one would often love to be able to trawl a wider pool – but simply because it alone has an index. Of course it is possible to use The Times’s index merely to check dates, which one can then follow up in other papers; or just as a chronology of events, without necessarily looking up any paper at all. Nevertheless, The Times is almost certain to be one’s first resort – not least because it is the one national paper which local libraries tend to keep, though usually on microfilm. The only place I know where one can still read the old heavy bound volumes is the private-subscription London Library in St James’s Square.

No responsibility

Of course, this development is not all for the worse. The newspapers of the 1950s were factual so far as they went, but extraordinarily dull: they tended simply to reproduce official statements with little questioning or analysis, reporting very little of what we now know from diaries, memoirs and the public records (accessible after 30 years) was really going on. Sensitive matters well known at Westminster were referred to only obliquely, if at all, in the public prints. The decline of deference in the 1960s produced an explosion of investigative reporting, speculation and informed comment – in all the papers – which furnished historians with a goldmine of vivid contemporary material undreamed of by their predecessors. My own experience over several books reflects the change.

When I wrote a biography of Nye Bevan – 20 years ago now – I found that The Times for the 1940s and 1950s was a paper of record but little more. The popular papers – above all at that time the Daily Express – were livelier; but they had no index, so all the difficulties described above applied. Only by using The Times’s index could I hope to locate the Express’s coverage of an event such as the notorious speech in 1948 in which Bevan described Conservatives as “lower than vermin”, provoking furious outrage in Fleet Street. There was one other way. In those days people used to collect press cuttings. There was even a substantial if somewhat random collection at Colindale, deposited I think by the Daily Mail; but its existence was not well advertised and I understand that it is no longer kept on site. The Labour Party also had a collection, held at the then party headquarters in Walworth Road but now, I believe, at the Labour history archive in Manchester. Newspapers, too, have cuttings libraries – the essential standby of many a scissors-and-paste article – but these are not usually accessible by members of the public.

When I moved on to write about Ted Heath I found The Times of the 1960s and 1970s transformed: indeed both as reader and researcher I look back on those decades as a golden era. The paper still took seriously its role as a paper of record, but carried in addition a wide range of intelligent analysis and argument from all parts of the political spectrum. Indeed, the central debate of the 1970s – the choice between monetarism and incomes policy – was largely conducted in its columns, with the economics editor, Peter Jay, an early monetarist and the editor, William Rees-Mogg, an important convert. For that period at least, leaning heavily on The Times did not give an unduly unbalanced or distorted view.

The same could not be said when I came to research the second volume of my biography of Margaret Thatcher, dealing with the 1980s. From 1981 The Times became at once less reliable as a paper of record and increasingly partisan: it was now just one more paper in an increasingly cut-throat market, no longer to be read with any special respect. Even its index ceased to be unique, though it remains more accessible than those of The Guardian and Financial Times, which are not generally held by libraries. Fortunately, so far as Mrs Thatcher herself was concerned, I was immensely helped by the timely publication by Oxford University Press (assisted by the Thatcher Foundation) of a CD-Rom comprising her complete public statements – every speech, every press conference, every interview and phone-in, every passing remark picked up while visiting a factory – amounting to some 14 million words. (“My whole life,” she marvelled at the launch, holding the disk between thumb and finger, “in four inches of plastic!”) That was an invaluable resource, meticulously edited, which reduced my reliance on newspapers; but there is no equivalent that I am aware of for any other subject.

But of course technology is making new strides all the time. Not only is The Times’s index up to 1980 on CD-Rom, but the entire text back to 1795 is now available in electronic form in most public libraries; a recentlyconcluded agreement allows any library member to log in and search it at home, free, just by typing in their library number. This is wonderful; but the effect is to further reinforce the unhealthy primacy of The Times as a historical source. Fortunately, Colindale is also digitising the text of some other 19th century newspapers; and some papers, such as The Scotsman, are digitising their own back-archive. The future is, in one sense, secure since most newspapers are now published online as well as in print, with a search capacity ensuring the next generation of historians will not need an index to find their way to what they want.

All available online

A final worry is how long all this material will remain online. Will it be preserved in any physical form? Already there are concerns about the durability of microfilm. Electronic archives are even more fragile. At the risk of sounding like a grumpy old Luddite, there is something very reassuring about those bound volumes in the London Library. Historians of the future may miss them very badly.