After a long break, about 30 years in fact, Manhattan is suddenly full of architecture again. The tragedy of 9/11 switched New Yorkers on to the power and presence of the buildings that surround them, to their importance in the psyche of the city. Decades of dumb, extruded skyscrapers gave way to a sneaky seduction by sophisticated international designers. But all the action so far has taken place in the rarefied, super-rich world of architect-branded condos. So the opening of an important public building, the New Museum, has caused a bigger stir in the city than any building I can remember. The first purpose-built museum downtown, the New Museum is a striking and serene addition to the rough and ragged profile of the Bowery, Lower Manhattan’s most tenaciously ungentrified street.

The gallery is a series of stacked boxes, piled up as if by a toddler in a slightly irregular manner and sheathed in a diaphanous veil, which turns out to be nothing more than expanded metal mesh. It is a thing of lightness and luminous beauty. Built on a former parking lot amid homeless hostels, grimy missions and furniture stores, the teetering tower is a strange riposte to the solid, bricky, industrial fabric that surrounds it, a reaction perhaps to the omnipresence of the industrial chic of the parasitic galleries that have invaded the city’s toughest spaces, from Chelsea to SoHo.

The delicate, anaemic architecture is both distinct from and a distant relative of the approach taken at the Museum of Modern Art in its eye-wateringly expensive ($800m) 2004 extension by Yoshio Taniguchi. Less minimal, less slick, less corporate, less luxurious but aspiring to a similar lightness and translucence, the New Museum is also the work of a Japanese office. Sanaa (the collaboration of Kazuyo Sejima and Ryue Nishizawa) is the fastest rising star on the international scene. Its ghostly, net-curtain architecture has seduced the art establishment, its style is subtle, sophisticated, almost not there – a quiet but powerful response to the overbearing fashion for self-consciously sculptural, iconic forms. Its works include the ethereal 21st-century Museum of Contemporary Art in Kanazawa, the Kunstlinie Theatre in Almere in the Netherlands, a seductive office for Novartis, the pharmaceutical company, in Basel and the seemingly weightless Glass Pavilion at the prescient Toledo Museum of Art.

The stacked boxes float, slightly disconcertingly and rather beautifully, over a wholly glazed ground floor that exposes a brilliantly lit, rather clinical foyer, shop and possibly the least cosy café in the world. The art starts in the first box on the first floor. The space is simple, cheap, unobtrusive, perhaps a little oppressive. The reasoning behind those shifting boxes becomes clear in a strip of skylight formed by the setback of the box above. The lateral movement of the boxes as the structure rises allows each gallery a crack of sky. It is a lovely device, a simple response to a practical need that governs the form of an urban infill, which rises to become a wobbly and paradoxically anti-monumental ziggurat.

The light is, however, not quite enough. The galleries are generous in proportion, their language a stripped exposure of construction techniques, but these little fissures of light illuminate only a single edge. The predominant feeling throughout the galleries, and in the claustrophobic stairwells, is of over-bright artificial lighting, a brilliant fluorescent glow of hospital intensity.

On the inside, this is a gallery that is serious about art. It eschews over-expression and over-detail. At a few points, windows open the building out to the skyline and those hypnotically irregular roofscapes of water towers and fire escapes but these are treated as little events (breath-catchers on a multi-storey stair ascent) and kept away from the art. The inaugural exhibition, Unmonumental, a big, international and cross-generational collage show, is flawed in its omnivorous blend of media and material. It has real problems hanging together but it is easily and comfortably accommodated across the floors. For any faults the building may have, the galleries here are flexible and low-key, and fade behind the art just as they should.

The galleries become more intimate as you rise through the levels and finally reach the seventh-floor “Sky Room”, the zenith of the building. An entire wall of glass, plus a bit around the side, reveals an epic view across the rooftops, downtown, over the Lower East Side and across the water to Brooklyn. This is not the classic midtown skyline of the Empire State and the Chrysler but the scraggy irregularity and the gritty realism of crime films, a version of the city that lives in the memory arguably even more powerfully.

When I was there a crowd of people stood mesmerised by a huge snowstorm and a sky as grey and heavy as lead that split to reveal piercing sunbeams of baroque intensity. The room was deathly quiet and haunting, filled with a hip, young mix of locals and art tourists. It was the most poignant example imaginable of the power of the self-image of a city. Manhattan appeared as if on a panoramic screen; the city became the subject. It felt as if the whole ascent had been directed towards this moment, in which the museum melted away and the everyday made magnificent displaced the art to envelop us in its grimy embrace.

It was hard to leave that room. Some seemed almost overcome, slumped in chairs, overwhelmed by the sky and the sheer filmic presence of the city. By the time I tore myself away, twilight was descending and the building was beginning to glow against an urban backdrop that remained resolutely dark, in every conceivable way.

At night, its curious form appears less three-dimensional, becoming a theatrical flat, a billboard advertising nothing. This sudden change makes you aware that this stack of boxes carries out the tricky feat of slotting into the Bowery yet evoking the city’s skyscrapers. Those setbacks and steps, those elongated ziggurats so familiar from structures ranging from the Rockefeller Center to the Woolworth Building (the shape was introduced to allow light to reach increasingly canyon-like streets) are remembered here in the tapering pile of shifting boxes. They are even parodied in the single overhanging box that gives a calculatedly precarious and unmonumental appearance.

The New Museum vacated cramped premises in SoHo for the Bowery and immediately you can sense the changes afoot. The last chunk of downtown to resist gentrification is about to surrender to the realtors. Already the self-conscious profiles of architect-(over)designed condos are puncturing the skyline, hovering, eager to impinge. For the moment, though, this pristine chunk of architectural intelligence looks all the better, all the more striking, in a bit of resolutely real city.

New Museum, 235 Bowery, New York
www.newmuseum.org

Edwin Heathcote is the FT’s architecture critic