The white woodframe house stood alone near the top of a hill in northern California. What was odd about it? Something was. Maybe it was the hint of gothic belying its innocent whiteness. Maybe it was the silence. Or maybe it was what was missing. There were too few crows. There was no playground with a climbing frame. And if you looked downhill to the south, you didn’t see a long street leading to the centre of a small fishing town, you just saw more rural emptiness.

Yet this was the schoolhouse from The Birds. I and my significant other were looking at it, goggling from our rented Chevy, having tooled up from San Francisco for no better reason than to check out Bodega Bay, nominal setting for Hitchcock’s aviophobic super-shocker. What better reason do two film nuts need? On the way we had visited – 20 years after James Stewart and Kim Novak – the Muir Woods giant redwood tree that gloomed and towered in Vertigo. Since then we had stopped off in Bodega Bay itself, a barely recognisable huddle of stores and quaysides, with absolutely no view across the water of a curve-round shore with a bird-imperilled house. (That’s what trick photography can supply in a movie.)

Then by pure accident, a mile or several out of town, we found the schoolhouse from which the crow-attacked kids ran downhill. In the movie they ran a few hundred yards straight into town. But even displaced, this stumbled-on lodestone of memory and association was inexplicably powerful. Black magic, white magic. A trip through the looking glass. A jigsaw puzzle to be assembled.

Why was this sighting so affecting? Was it that fact had broken out of fiction? Was it that imagination had birthed a visible, touchable reality right there in front of us?

It wasn’t the first time – let’s call it the thousand-and-first – that “ordinary” places have got to me. Most times, though I’m a movie critic, this has nothing to with movies. In the right circumstances a place just burrows into the soul like one of those river flukes you don’t see before they nest in your liver.

I am talking about spirit of place. Genius loci. We all recognise it. Some may be affected by it more than others but I suspect not. I suspect we are all pleasurably and powerlessly at its mercy.

I don’t mean noted or listed beauty spots, the lingua franca of the travel brochure. Of course Venice is beautiful, and Machu Picchu, and the Golden Gate Bridge. But I’m talking about places veined on the map of your being: places that do for your psyche what the madeleine did for Proust after being dunked in his mug of tea.

For that’s it. The place, like the madeleine, is nothing without the dunking. It’s the chemical action between a place and a life, or something in the experiencer’s life. (The Birds is a film I love, venerate and keep revisiting. I think of it as a friend.) Here is another instance.

For years I visited my parents, who lived near Henley-on-Thames, by driving out of London on the M4 motorway, a featureless journey along dinning Tarmac. But soon after my father died and was buried in a country churchyard near his village, I discovered another route to the same destination. It was as if it had been “given” to me. The route was no less direct, yet it had a new dimension of enchantment. It didn’t touch a motorway. It skirted the Thames after Hampton Court, drove by Windsor Castle, re-found the river before Runnymede, went through the park, bumbled bucolically through Bray. Time passed so scenically that I felt I was hardly making a car ride at all, in the sense of an ordeal by traffic and pounding asphalt. It was like finding a wormhole in terrestrial space.

Then – a less familial but no less forceful happening – there was the Eugene O’Neill house in Connecticut. I studied O’Neill once for a PhD, never completed, so he became a kind of pal. He took up squatter’s rights in my life and mind. I tried – as a doodling imagination does in hindsight – to help him out of his drinking, to commiserate over his broken romances, to tut-tut with him over his daughter’s marriage to Charlie Chaplin. (How he hated that. An ageing British vaudevillian for a son-in-law!) So when it dawned on me, during a fall vacation in New England that had ended up in Mystic, Connecticut, that the O’Neill home was nearby, across two rivers, I went berserk. This time my significant other, carrying no O’Neill baggage, wanted nothing to do with my madness. So I drove to the place during a minor hurricane, had a janitor (who happened to be passing by) open the closed-for-winter building, and wandered round this woodframe mansion still haunted by its other life as – of course – the setting for Long Day’s Journey Into Night.

I realised how subjective this experience was, how personal, how obsessive-compulsive, by the glazed politeness of people I later related it to. “Oh, the Eugene O’Neill house. Really? How interesting.” Yet for me, the joy poured out of my eyes and ears for days. All for an ordinary-looking suburban structure you wouldn’t glance at twice.

What does it all mean? It means places have a matchless quotient of what Keats called negative capability. They soak up association as nothing else does. They hoard emotion. Objects do this too but in a more circumscribed way. Object-fetishism is more analysable, therefore less magical. We all know why Garbo in Queen Christina wanders round the bedroom that was once a love-nest, fondling or hugging the contents. She’s honouring a memory: a man and a romance. Proust’s madeleine is another single object, though already moving towards a greater complexity because of that act needed (the tea-dunking) to catalyse memory.

Places are by a quantum leap more complex and polyharmonic. Around the core of a definable link with the experiencer’s life there swim mysteries and hauntings, things that he, she or we cannot know of – other people, other lives, other visitors – and that enrich the power of place as extra instruments make a symphony richer than a sonata.

Look at two places in literature. Manderley in Daphne du Maurier’s Rebecca is made magical from the novel’s first sentence, that haunting, perfect Alexandrine. “Last night I dreamed I went to Manderley again.” The mansion is possessed by so many ghosts that it’s a living coven in brick and stone. (Or it is until burned to a crisp.) The brooding husband, the jealous housekeeper, the unforgotten first wife, they are all there. No wonder the house must be incinerated for life to move on.

Then there’s Varykino in Boris Pasternak’s Doctor Zhivago, just as magical for an opposite reason. It is its emptiness that is fabulous, strange, illimitable. It’s an abandoned estate in the snow where the spirits of the true owners – those “disappeared ones” of the revolution – bark like the distant wolves and where Zhivago’s and Lara’s love camps impermanent and imperilled like a besieged town. Varykino becomes the book’s image of romantic love, a place so precious, endangered, unsheltering that every night is a victory for the human heart over the inhuman collective that is, or is becoming, the rest of Russia.

Manderley and Varykino are often described as “characters” in their novels. But they are more. Each is the defining element in which the characters live: their air and ocean, refracting and shaping them, and refracted and shaped by them.

Places become precious not through their intrinsic beauty or historical interest but because they collide with key moments or moods in our lives. What is the last thing a lover says to a lover – or the last loving thing! – when they split up? Don’t take your new lover (he or she says) to the place that was precious to us. Whatever it was: a pub garden, café, the corner of a park. These memories are intimate keepsakes. That’s why Manderley declares war on the second Mrs de Winter. It’s Rebecca, the first wife’s, domain: the place where she and Maxim were happy. Trespassers will be persecuted.

These whirlpools of psychic ownership are powered by deliriums of association. These deliriums, in turn, are a kind of synaesthesia. Just as some people see numbers as colours, or experience musical sounds as scents, places and emotions, with almost everyone, can interfuse.

More weirdly – it happens to me at least – I keep finding that places rhyme inexplicably with other places. A stretch of swampy, willowy embankment on the A259 between Bexhill and Eastbourne always evokes a commensurate stretch of scenery near the bridge to the Keys in mainland Florida. Why? Goodness knows. A bend in a road in Kent replicates a particular bend in a road in India. It’s as if they have the same DNA or have been cloned one from another.

The two locations, in each case, seldom have any special resemblance. So what has happened? I must have had the same thought or emotion at both places and this sutured them together.

In turn, a place can be used to prompt or re-stimulate an emotion. In a small Kent village there’s a white Georgian-style house where Joseph Conrad lived. It nestles in a beautiful walled garden next to the village church, with less than a dozen other houses in view, crawling up the opposite hill. This house has become the turnround point for a drive on which I take my mother, now 92. It goes through a series of lanes and parklands almost ridiculously picturesque. Now we are in a country alley squeezed between woods and fields where, if you stop with an open window, horses will poke their heads through, demanding sugar lumps. Now we’re breezing along on a high park road with a view across sheep fields to an imposing Queen Anne mansion. Now we’re dipping into the village, where Conrad and his ghosts are already gearing up for the writer’s 150th birthday.

In my head this drive has become – or I suspect it has – a way of holding time still. Take a calendar countryside and infuse it with eternal recurrence. By that means my mother gets to forget she’s a nonagenarian (though she seems rather proud of that) and I get to forget that most of life is fleeting, elusive, uncontrollable and cannot be magicked like this into a transcendent, timeless scenic route.

At the same time, by the principles I have been setting out, this drive is a bit of a cheat. For places of the heart shouldn’t depend on prettiness. True places of the heart are inoperably subjective. They are so virally filled with emotion and association that it doesn’t matter if they are picturesque. They get to you, they live in you, anyway.

I know the place I mainly mean. Cross a railroad track in Calabria, Italy, to a closed-season beach where a single kiosk vendor sells cheap snacks in a maze of concrete. There are tables, benches and wastebins. This deserted spiaggia, attached to a small Adriatic town, could almost come from an end-of-the-world movie. It’s blown about by emptiness. It’s not pretty, nor even appealingly quaint. But year after year I visited it, or passed through it, with the right person at the right time.

We had knocked off work, freedom stretched ahead and we were going somewhere warmer and nicer. But, of course, nothing was warmer and nicer than this. It was a shared nowhere. It was impregnable by its ordinariness. It was vast with that esoteric exclusiveness by which two people claim a compass point as their own. “We’ll always have Paris,” said Bogart to Bergman. That’s good. But I feel I can do better. (so does everyone at the best moments in their lives). I can say, “We’ll always have – Blank.” Because you won’t get the name from me. And you mustn’t even try. Let’s just say, it’s in southern Italy, it is nothing to write home about, but it is my Paris, my Varykino, my Manderley.

Nigel Andrews is the FT’s film critic