Oh BELGIUM, man! was Zaphod Beeblebrox’s oath of choice when his two heads were pushed right to the edge. But if The Hitchhiker’s Guide to the Galaxy meant the curse to suggest rain-soaked tedium relieved by stabs of edgy peculiarity, it got Belgium wrong, the tedium bit anyway. This is the country that gave us Bosch and Magritte and, right in between, the feverish thing that was James Ensor. By turns lurid, lyrical, mysterious, sophomorically satirical, intimate, raucous, cerebral, macabre, tender, narcissistic, suicidal, iconoclastic, reverent, supersaturated and washed out – and that’s just in the first 15 years of a half-century career – Ensor is the figure no conscientious chronicler of the birth pangs of modernism can afford to overlook, but also the one whom no one has any idea what to do with.

So good for New York’s Museum of Modern Art for giving him the first major show in a very long time, even though the curatorial effort to plug him into the genealogy of modern art turns out to be a futile enterprise. He’s catnip for the ism-hunter, since he could do the lot. So the wall-captions and the catalogue assiduously nail this and that picture to the ism du jour – impressionism, expressionism, realism, surrealism, slathery tachism and pretentious symbolism. But his sensibility was as twitchy as a bat, swooping out of the air and vacuuming up whatever took his fancy on any particular Tuesday.

His whole career, at least the interesting two decades of it, was one long carnival guffaw at the higher seriousness of modernism. You can almost hear the kitschily raffish self-portrait – Rubens on absinthe, crowned by a Quentin Crisp snapbrim trilby, primroses bursting from the hatband – squeal with giggles from the wall at the solemnity of the installation. Oh Belgium, man!

And not just Belgium but Ostend: the place you went to for marine calm when Brussels was just too much fun. Which is what Ensor did after a short stint at the Ecole des Beaux Arts, where he rehearsed a lifetime’s habit of getting on everyone’s nerves.

King Leopold’s Brussels, grown fat on a deadly combination of voracious African imperialism, Liegeois coal and local waffles, was a pin cushion for the avant-garde. Its luminaries – the group who called themselves “Les Vingt” and their critic-patriarch Emile Verhaeren – were alert to anything, within and without their borders, that would shred the overstuffed upholstery of bourgeois academicism. It was Verhaeren who wrote the first gushing review of Van Gogh and who wanted Ensor to exhibit with Les Vingt. He became one of their number but was never a team player. So in 1880, when antsy modernists were hungering for a spell in the South Seas or the boulevards of Paris, Ensor returned to his native town of Ostend, from which he would seldom budge for the seven decades of the rest of his life.

Which is not to say that Ostend was a dull backwater of a middling province of a Lilliputian kingdom. It boasted a small-time bohemia, oompah and pickled herring on the promenades, chalky pierrots and madcap Punchinellos; it had beer and bathing huts and curio shops, one of which was owned by Madame Ensor, James’s doughty Flemish mother. His father, not so doughty, was an Englishman of means who no longer had them, and had wanted to plant a theatrical sense of himself between the salt marshes and the slate-coloured sea. In their artist son, the two parental tempers – morose and exuberant – bounced off each other, with mood-swinging results.

The 20-year-old James moved into the loft above the novelty shop and plundered its inventory, especially its grinning or wailing masks, to brilliantly histrionic effect. It wasn’t an especially profound or original insight, the Ensorian perception that the grimacing carnival mask might express the Real Persona while the flesh and blood face was merely the mask of social convenience, but only Goya and Daumier had made the masquerade such a penetrating genre of psychological portraiture.

Did the obsession with masks actually produce compelling art? The answers in the MoMA show will surprise anyone who thinks they know the mischievous Monsieur Ensor well. “The Scandalised Masks” (1883) is a throwback to the 17th-century genre painting of Adriaen Brouwer (for all his antic iconoclasm, Ensor was a great archivist of the Netherlandish tradition), with the startling difference that instead of two boers in a tavern, a half-doped snout-nose looks up to see a clogged and bonneted woman, sinister in dark glasses, coming through a door, clasping a wind instrument. Her gesture, also drawn from low-life painting, is unsubtle, the invitation more ominously castrating than seductive.

Much dalliance with intimiste impressionism follows, with Ensor freckling his light through the tufty woollen gloom of Ostend parlours. But one stunningly weird picture, called, as if to provoke the burghers, “Lady in Distress” (1882), promises something unsavoury: a Sickert of the dank seaside. She’s not so much In Distress as completely out of it: eyes half-closed, body sunk into the featherbedding as if awaiting a shroud. One window is draped with a sallow fabric; at another, the curtain is pulled back, its swag hooked to the wall. But the light coming through the exposed pane is painted with a clotted flake-white impasto, while the backlight behind the closed drape is suggested by the thinnest of paint stains, scraped back with palette knife and brush handle. Light is dark and dark is light. Welcome to Ensor’s univers pervers.

Occasionally Ensor takes it easy, and the results are breathtaking. The enormous “Rooftops of Ostend” (1884) is Turner meets Jacob van Ruisdael, yet somehow translated into an idiom that was all Ensor’s own. The sky, cerulean and pink, creamy and dove-grey, covers seven-eighths of the canvas, and is passionately worked with the palette knife and broad brush, while the crowded roofs below, in their contrapuntal dance of planes, anticipate cubist townscapes.

But Ensor didn’t give a hoot for where he was supposed to stand in the face-off between tradition and modernity. Another revelation of the show are the enormous, monumental charcoal drawings, with their multiple quotations from Ensor’s art-god, Rembrandt. Ostensibly scenes from the New Testament, the artist sets them in a Belgian carnival, complete with high-hatted bandsmen. And if the loss of the Son of God amid the seething throng seems a modern caprice, Ensor knew very well that it looked back to swarming pictures by Bosch and Bruegel, in which the indifferent brutality of the crowd engulfs the redemptive presence of the Saviour.

Ensor himself never really felt redeemed, and after a while the jokes in which genre figures, and his own self-portraits, are replaced by skeletons, wear even thinner than the rattle of bones. His mind was too frantic, its demonic energy never crystallising around a painterly language in which form and content marry without acts of self-conscious main-force. He means to jangle but in the end the cacophony enervates rather than illuminates.

Every so often, though, there are treats when he seems to reach back to the boy who must have watched the waves roll in to the beach: a little version of the Battle of Waterloo, with numberless toy-soldier riders hurled against each other, and, at the other extreme, a cartoonish beach scene, complete with smiley-faced Mr Sun, an array of bathing huts, the dip and bounce of beach bums, two men snogging in the foreground – the whole “ooh missus” malarkey of the thing a little cartwheel of pictorial joy. And you think, this may not be vanguard art, but it has the quick of life in it.

‘James Ensor’, until September 21, Museum of Modern Art, New York; www.moma.org

Simon Schama is a contributing editor of the FT