With the muscle memory of the talk-show host still strong, Clive James holds out an introductory arm and ushers me into the library of his Docklands flat. He delivers an opening spiel that sounds almost as if it has been said many times before: “This is my bachelor pad, but Cambridge is where the books really are,” he says all the while pointing to a selection of many books in six languages – including Russian and Japanese – that he’s managed to acquire since leaving Sydney in the early 1960s.

At 68, James has finally reached the age that he has looked for the past 20 years. He’s still heavy-set and paunchy but gone are the tight television suits that he used to be funnelled into like an overgrown schoolboy, or Alexei Sayle. These days, with his dark-rimmed specs and existentialist uniform of black shirt and trousers, he could pass for a professor of English literature.

Previous interviewers, expecting to meet a sun-drenched larrikin, have been disappointed to find James in a state of wintry pessimism. Today, perhaps, because he’s fed up with being portrayed, in his words, as a “self-questioning, paranoid sad-sack”, he’s doing a fine impersonation of a man who is giddily upbeat. But it quickly becomes clear that there is a reason why the endorphins are pumping: James finally has a hit in America. His latest book, a 900-page cultural history of the 20th century, Cultural Amnesia: Notes in the Margin of My Time, has entered the bestseller lists in a land that is entirely unacquainted with his TV persona. James, who has just returned from a promotional trip to New York, admits he’s been guilty of the writer’s secret vice – checking his own sales figures on Amazon: “I moved up 10,000 places. Every writer does it and it’s a fatal, fatal thing.”

Cultural Amnesia is an A-Z compendium of the century’s greatest thinkers, artists and philosophers comprising more than 100 different essays – culled from the notes in the margin that James has been making for 40 years. But instead of supplying potted biographies, these figures – from Louis Armstrong to Ludwig Wittgenstein – are used as the starting point for some gonzo intellectual jamming. An essay on F Scott Fitz- gerald becomes a rumination on how far great writers are influenced by the prose styles of their heroes. And an entry on Charlie Chaplin – via a detour on Albert Einstein – explores the difference between artistic and scientific knowledge. The effect is rather as one might imagine the caffeinated froth of a never-ending conversation in a pre-war Viennese café. But, emerging from the recherché references and indulgent asides, the book has one Big Theme: Cultural Amnesia turns out to be a defence of the arts and liberal humanism in all their complexity as a bulwark against the totalitarians and postmodernists who look for the “dream solution” that will reduce life to a formula. “You’re claiming a lot if you think you’ve accumulated the kind of wisdom that you can pass on,” says James. “But it’s an instinct I feel. What else do you get out of 70 years of life except a certain distilled knowledge?”

The moral centre of the book is the personal conduct of both obscure and high-falutin’ figures when faced with Nazism and communism – and, critically, whether they collaborated: “Sartre was as clever as a man can be. But somehow it wasn’t [the sort of] brilliance that it took to realise the kind of enemy totalitarianism was – and how the left version was just as bad as the right,” says James. “I have a feeling that he did get it but he didn’t care. He had all the information about the gulags but it didn’t affect him.” Pitted against Sartre is James’s hero, Camus, who worked for the Resistance in occupied France: “He’s the frail man I’d quite like to be. He was so charming, the women loved him – and he was also right on the big simple thing.”

The window in the diary that allowed Cultural Amnesia to get written was created by James’s sudden disappearance from television in 2001. Throughout the 1980s and 1990s, he was one of the most famous men in Britain – a wisecracking prime-time fixture with shows such as The Clive James Show, Saturday Night Clive and The Late Clive James. He spent years flirting with the likes of Jerry Hall and Elle Macpherson, unearthing clips of Japanese game-show contestants stuffing tarantulas into their Y-fronts, and introducing the off-key charms of Cuban club singer Margarita Pracatan to the nation.

By the millennium, with a new generation of rapiers such as Jonathan Ross and Graham Norton on the scene, James was shunted off into Monday Night Clive. When his Postcard from Havana remained on the shelves for 18 months after filming, he took the hint and gave up on TV completely. He claims to welcome no longer being part of the wallpaper of people’s lives. “It’s much more comfortable in the street now – and I honestly don’t mind that. I get hassled much less and when I do, it’s normally somebody mixing me up with Clive Anderson.”

I ask whether those years of lucrative TV cheques were a bargain with the devil that saw him lose some of his best writing years? “I never look down at my work on TV – even the journeyman work. First, it was too beneficial in making prosperity possible for my family. They made it possible for me to do the Postcard shows – my best work on TV. And I would not have had my publishing career without my TV career.”

These days, safely beyond the brutalities of commissioning editors, he is pouring his energies into his multimedia mausoleum, Clive James.com – run from his flat with the help of a young French assistant, Cécile Menon. On the site – entombed amid links to old interviews with the Spice Girls, yellowing speeches given to the BBC governors and his poetry – can be found his latest series, Talking in the Library, which is also appearing on cable arts channels round the world.

In the show, shot in the library in which we are sitting, James swaps salon talk with old mates such as Martin Amis, Jonathan Miller, Deborah Bull and Cate Blanchett for half an hour at a time.

Away from the digital byways, he returned to a mass audience on Radio 4 this year when he inherited Alistair Cooke’s old Sunday morning slot with his weekly essay A Point of View – a typical Jamesian ramble through subjects ranging from media portrayal of torture to JK Rowling to wheelie bins. “I love it,” he says with feeling – obviously basking once more in an audience counted in the millions. “It takes me back to the time when I was a TV critic preaching a sermon every Sunday morning. It’s wrapping yourself in a toga, standing up in the agora and saying your piece.”

This hogging of the limelight “is all I’m fit for”, he says. “I’ve never been good at holding down a job.” His early scrapes in the employment market – from fabricating market research to hiding in the toilets with a book instead of working – are treated uproariously in his bestselling memoirs. But the tone darkens when he admits that, in his early 20s and penniless in England, he considered suicide: “I still have twinges about whether I should have written that episode because I’m a great believer that you shouldn’t show too gloomy a face to the young,” he says quietly. “I was at a time of despair. The proof that I was getting ready to jump off a cliff or stick my head in an oven – that I was serious – was that I was giving my books away. It was largely because I was lost, I had no outlets, and I wasn’t expressing myself. I wasn’t doing what keeps me stable now – which is having a stage and a platform.”

The writer’s regime he has settled on is defiantly anti-social. During the week, James lives a life of solitary bachelordom in the apartment for which only he has a set of keys. He subsists on sardines and over-boiled eggs, returning at weekends to Cambridge to see Prue Shaw, his wife of 40 years and a modern languages lecturer, and his two daughters. “This is a student existence. I’m some kind of shambling slob. I don’t have any toys and few luxuries. But there is a neatness at the centre of it. I keep going at my work until it is clean.”

Now that James has broken his 80-a-day nicotine habit (“I used the wheel hub of a British Bedford van for an ashtray”) and left his days as a hard-drinking denizen of Grub Street behind, the only vice that he’ll now admit to is work. And the roots of this obsession aren’t hard to find. James’s father, an Australian soldier taken prisoner by the Japanese during the second world war, survived a prisoner of war camp but died when the aircraft taking him home crashed. At the time, his only son was just six: “I’m a great believer in using time productively – it’s probably psychological because my parents were robbed and I’m paying them back – I’m leading their life.”

Despite producing four volumes of memoir, James barely mentions his family at all, bound until recently by a pact to keep them out of the press. But his eldest daughter, Claerwen, a portrait painter, gave a wry interview last year when she said she had “no reason to believe” that James was “involved in the practicalities of my upbringing”.

“She’s divine, of course,” says James, aware that he’s treading on thin familial ice. “She sees me a lot now but I didn’t find children very interesting – it’s true. I was too busy. I’m terrible on holidays. I’ve just got back from one on which I was universally unpopular. We had the third generation with us this time and I built the sandcastle. Typically, I overdid it.”

James gallops through his plans for the next 10 years – there will be a sequel to Cultural Amnesia, he’s working on a novel on the Japanese-Australian war and is writing an operetta about the tango. Is he feeling the press of mortality?

“The sense is that I’m running out of time. It’s a natural term and I’m sorry I’m reaching the end of it. I’ve got something done. It’s just that I’d like to do more.”

Since we’re on the subject of endings and I can see him surreptitiously glance at his watch, ready to finish the interview, I ask whether he is really, as he has idly speculated, planning to return to Australia, living out his days fishing with the other old men in Sydney Harbour?

“I don’t know where I’ll leave my bones,” he says, “but the last thing I’ll do is write a poem, not catch a fish. And I can do that anywhere.”

The many voices of Clive James

Literary critic:

Shakespeare’s language is not pure. Even at its most exalted, it declines to be exclusive. It switches between one level of decorum and another as an electron shifts orbits without crossing the space between.

From ‘The Meaning of Recognition’, published by Picador in 2005.

TV critic:

Every week I watch Stuart Hall on It’s A Knock-Out and realise with renewed despair that the most foolish thing I ever did was to turn in my double-0 licence and hand back that Walther PPK with the short silencer.

From ‘Visions Before Midnight’ published by Jonathan Cape in 1977.

Pop lyricist:

The heroes return through the Sunset Gate

But their faces are never the same

I have no idea why their eyes go cold

And the young among them already look old

But high behind them the sky’s aflame

In the flickering hour of their fame

From ‘Sunlight Gate’ on “Driving Through Mythical America” (1971), one of James’s collaborations with musician Pete Atkin.

Essayist:

Even before I met her, I had already guessed that she was a handful. After I met her, there was no doubt about it. Clearly on a.hair-trigger, she was unstable at best, and when the squeeze was on she was a fruitcake on the rampage. But even while reaching this conclusion I was already smitten.

From I Wish I’d Never Met Her, an article on Princess Diana for The New Yorker.

Poet:

As any good poem is always ending,

The fence looks best when it first needs mending.

Weathered, it hints it will fall to pieces — One day, not yet, but the chance increases

With each nail rusting and grey plank bending.

It’s not a wonder if it never ceases.

From ‘Status Quo Vadis’, a poem published in The Spectator in 2006.

Commentator:

When I say that I’ve had enough of celebrity status, I don’t mean that I am sick of the very idea. As it happens, I think that the mass-psychotic passion for celebrity – this enormous talking point for those who do not really talk – is one of the luxurious diseases that western liberal democracy will have to find a cure for in the long run, but the cure will have to be self-willed.

From ‘The Meaning of Recognition’

Tango dancer:

I’ve only just started having fun after four years. It’s worth the effort, though. It’s the world’s most beautiful dance.

Extract from a BBC interview

‘Cultural Amnesia: Notes in the Margin of My Time’ by Clive James is published by Picador, £14.99