Politically incorrect

May 8 1969: the Labour Cabinet, under Harold Wilson, was discussing the Industrial Relations Bill, which was designed to curb the power of trade unions a decade before Margaret Thatcher came to power.

It was an extraordinary occasion. Jim Callaghan, home secretary, was leading the internal opposition to the bill when his colleague Richard Crossman finally lost his temper. “We have got to sink or swim together,” said Crossman.

“Not sink or swim,” replied Callaghan. “Sink or sink.”

“Why can’t you resign if you think like that?” said Crossman. “Get out Jim! Get out!”

There was an awkward silence. Then Callaghan muttered: “Of course, if my colleagues want me to resign, I’m prepared to go if they insist on my going.”

According to the prime source for this exchange, Callaghan “had been punctured. He hadn’t responded, he had crawled and it was quite a moment.”

The source was Crossman, whose diaries caused a phenomenal rumpus when they were posthumously published in 1975. He was backed up by another cabinet diarist, Barbara Castle, though in her account the wording differs slightly. A third, Tony Benn, describes the meeting as “remarkable” but fails to report the exchange.

Scribble, scribble, scribble! Political diaries have become a British phenomenon, both in political and publishing terms. As one former minister advised a Cabinet incomer: “The first thing to do is look round the table and think, ‘Which of these bastards is keeping a diary?’ ” One well-placed source reports that at least three current Cabinet members appear to be taking more notes than would seem necessary for ministerial purposes.

The most successful can turn into a pension plan for their authors. As Mae West put it: “Keep a diary, and one day it’ll keep you.”

A fascinating new anthology, Events, Dear Boy, Events, edited by Ruth Winstone, shows it is even possible to write a history of the past hundred years through snippets from diaries. But the genre is far older than that. The template was created by Samuel Pepys in the 1660s, though it was not until the 19th century that anyone cracked the code of his shorthand and discovered the glory of it.

Generally, the notion of keeping a diary is now considered a bit old-maidish. And Winstone, who has edited diaries by Benn and Chris Mullin, reflects in a postscript: “The political diary may well have reached its natural end. A new technology has turned the centuries-old concept on its head: the blog, the Twitter and the thread.” And with publishers and newspapers enduring hard times, the earnings potential has diminished. For all non-fiction, both advances and payments for serialisation rights are reckoned to have halved over the past decade.

Then again, private diaries have many useful purposes: an aide-mémoire, a means of letting off steam, an act of personal homage to the value of one’s life. But the most interesting information is held by those barred from revealing the full story instantly. Britain is a country addicted to suppressing the truth, personal and political, so the insider’s record of events can provide a healing antidote of disclosure. Though government records are supposed to be revealed after 30 years, the most embarrassing documents have a habit of disappearing.

And, as evanescent electronic communications replace documentation, the diary could be more important than ever. Tsarist Russia supposedly consisted of “absolutism tempered by assassination”; British politics could be described as “secrecy tempered by diaries”.

Tory MP and diary writer Alan Clark with policeman in 1996

The most enduring diaries rarely come from the very top. The important thing is proximity to power. Pepys was a senior official at the Admiralty, and his diaries covered a tumultuous decade, the 1660s, and mixed close observation of great events – the Great Plague, the Fire of London – with an unexpurgated examination of his own life and frailties.

His first great successor, a century and a half later, was Charles Greville, clerk to the privy council from 1821 to 1861. Unlike Pepys, he was an aristocrat and was decidedly more discreet, about his own sexual activity and everyone else’s. But he was a wonderfully sharp observer and mixed a general tone of equable amusement with the occasional wasp sting. “Nobody was ever less regretted than the late King,” was his parting tribute to George IV.

Greville’s diaries were not published until 1874, nine years after his death. Queen Victoria, a published diarist herself, was livid: “An act in DISGRACEFULLY bad taste,” she wrote. And the three greatest diarists of the 20th century fitted a similar mould: “Chips” Channon (1897-1958), Sir Harold Nicolson (1886-1968) and Alan Clark (1928-1999). All were MPs; none of them got a sniff of the Cabinet; all had sex lives that would have startled Queen Victoria: the first two bisexual, Clark voraciously heterosexual. Significantly, all were extremely well-connected with access to the grandest dinner parties and the best gossip.

Though involved and interested in politics, they were never obsessed by it. Their diaries display a splendid breadth of interest but also a writerly introspection. And, though this is hard to judge, they appear to have at least started their diaries without any – or much – thought of publication, just as Pepys and Greville must have done.

Ion Trewin, who edited Clark’s diaries, thinks this is a pre-requisite for greatness: if the potential market determines the contents, it changes the relationship between writer and diary. “I can tell you to the week Alan realised they could be published,” says Trewin. “It was at the start of the Thatcher government, when he began doing his own footnotes.”

The Nicolson diaries really do seem to have been made public only because old age left him short of money for his own care. The diary was just a habit, he convincingly insisted to his son, “like brushing your teeth”. But what do we make of Channon, in the early days of his diary?

July 26 1935: “I feel caddish, even treacherous sometimes, keeping this diary from the eyes of my wife yet it is our only secret ... What is more dull than a discreet diary? One might just as well have a discreet soul.”

For connoisseurs of this subject, the Channon diaries retain a peculiar fascination because the unexpurgated version has never been published. It is thought to contain homosexual revelations and possibly juicy stuff about both King Edward VIII’s abdication and the Munich crisis. The rights have passed to his grandchildren and there is no sign of them agreeing to release them, though everyone involved is now dead.

It’s true that, once their initial charm wore off, neither Channon nor Clark would be first choices as desert island companions. “Being nice is not an asset in a diary writer,” says Trewin. Treachery can be, though. Churchill’s doctor, Lord Moran, breached medical confidentiality. Edwina Currie blabbed about her affair with John Major. Dick Crossman merely broke political convention, detailing his rows as housing minister with his department’s “utterly contemptuous and arrogant” civil service head, Dame Evelyn Sharp.

Whitehall did everything possible to prevent publication. And when it failed, the world Crossman described became the factual basis for the eternally popular TV series Yes Minister, with its depiction of politicians as well-meaning numbskulls and senior civil servants as Machiavellian opponents of change. But, as a popular shorthand view of British governance, the demonisation of the mandarins has been replaced by the version perpetrated by a current TV comedy, The Thick of It, in which everything and everyone is controlled by egomaniac press officer Malcolm Tucker, a character quite obviously based on Alastair Campbell, Tony Blair’s head of communications.

Campbell is an interesting and rather depressing case, since he controlled the message with unprecedented vigour while in Downing Street and is now doing so thereafter: his message – perpetrated in four successful volumes of diaries – that the saintly Blair was bedevilled by Gordon Brown, his irrational chancellor and rival, has become conventional wisdom. This is testament to the power of a diary, as opposed to memoirs, which are nearly always less readable and less credible. The sense that a diary has been edited but not rewritten with hindsight, is vital to its appeal. And, the Campbell diaries seem straight in that sense.

They also fit into another subgenre: the quasi-authorised version produced by a politician’s assistant. This was true of Sir Jock Colville’s much-admired account of wartime life with Churchill and the revelations about Lloyd George by his aide AJ Sylvester, who was thought by the Lloyd George family to have joined the traitor’s camp by saying too much about sex and money.

Richard Crossman in 1945. The Labour politician’s diaries caused a stir when published in 1975

There is some evidence that perceived niceness can be a help, at least in sales terms. The Clark diaries are a wonderful read but I know would-be buyers who refused to go near them because they thought he was a complete shit. By contrast, part of the appeal of the Nicolson diaries is that they create the sense of a gentle soul leading an enviable life, designing the gardens at Sissinghurst with his wife Vita Sackville-West between agreeable parties and a little light politics. The mild-mannered Labour MP Chris Mullin’s diary sold well, though it was neither nasty nor revelatory.

Several enthusiasts cite with approval Gyles Brandreth, a minor media personality and a junior whip under John Major, whose diary of his single term in the Commons, Breaking the Code, is a very jolly romp through the febrile fin de siècle Tory years.

Edwina Currie, a bestselling novelist but not a well-liked politician, had her second volume of diaries serialised last month in the Daily Mail. This week they were still languishing around No 16,000 on the Amazon chart. A similar fate befell her first volume, despite the juicy Major stuff. “I counted them all out,” said someone at the publishers after the books went out to the shops. “And I counted them all back.”

Iain Dale of Biteback, the publisher responsible for the second volume, is happy to bring out political diaries even if he expects a sale in the low four figures, which big publishers would scorn. “I really like publishing diaries,” he says. “A diarist puts far more of themselves into the book than any other author. You get a far more rounded view of them and very often your preconceptions can be shattered.”

The Conservative MP Keith Simpson is acknowledged at Westminster as the expert on this subject. Counter-intuitively, he singles out Harold Macmillan as a particular favourite, and an exception to the rule that the most successful politicians are too busy to write diaries, at least not good ones. “He was a very lonely man. I think the diary was his friend.” Simpson also owns Modernization of Conservative Politics, Diaries and Letters by William Bridgeman, who was briefly home secretary in the 1920s. “Very boring indeed,” he says.

No doubt the diaries will keep coming, the brilliant and the boring. But the potential authors in high places, the vengeful ones especially, might just care to reflect on the fate of the anti-Nazi head of the Abwehr, Admiral Canaris. A month before the end of the war, the German authorities found his diary. And they hanged him.

Matthew Engel is an FT columnist

PMs as seen by diarists

Ramsay MacDonald

“He said the National Government was misrepresented so much in the newspapers that the Cabinet thought they had better tell the public their point of view over the wireless. I said that of course they were able to do this by order, but I did not think it would be very advisable.” John Reith of the BBC, 1934

Winston Churchill

“At dinner he was very gay and sang old-world Cockney songs with teddy bear gestures. In the course of the day he said he had been mistaken over the abdication.” Noël Coward, 1943

Margaret Thatcher

“She is SO beautiful; made up to the nines of course for the television programme, but still quite bewitching, as Eva Peron must have been.” Alan Clark, 1980

Tony Blair

“He said he had the evidence and, well, he declared war on Iraq on behalf of Britain in support of America. Totally riddled with error, and he was standing there smirking and smiling. You’d think he’s just done something very clever.” Tony Benn, 2002

‘Events, Dear Boy, Events: A Political Diary of Britain from Woolf to Campbell’ (Profile Books, £25), edited by Ruth Winstone

Copyright The Financial Times Limited 2017. All rights reserved. You may share using our article tools. Please don't cut articles from FT.com and redistribute by email or post to the web.