Say improvisation and people think of beery stand-up routines, jazz riffs and scat singing or, if they’re into hip-hop, rapping and beatbox. Either way, the word suggests something instinctive, something brash and slapdash, a little coarse, perhaps, but often highly skilled – and something best heard through the haze of a smoky club.

What’s often forgotten is that improvisation, or extemporisation, once played an integral and universal role in music-making, and that it still finds expression in Indian ragas, Spanish flamenco and folk traditions the world over. In Europe the most fruitful period was the Renaissance and early baroque but an interest in improvised music faded over subsequent centuries. It would be wrong to suggest that it simply died out: cadenzas were popular in classical concertos and bel canto opera, although the majority of these were written, but improvisation proper (as opposed to circumscribed decoration) is now the preserve of specialist musicians.

One of the pioneers in this field is lutenist Paula Chateauneuf. Throughout her career she has explored the art of improvisation in early music and she aims to give it wider public exposure next month with Take The Risk, a weekend festival she has curated for the Southbank Centre in London. “It’s been a long time coming because there aren’t that many people who are willing to get on stage and play from incomplete scores,” Chateauneuf says. “It takes a particular kind of person who wants to delve in and spend time learning the musical language.”

Backed by a fellowship from the Arts and Humanities Research Council, Chateauneuf has assembled an eclectic range of early music virtuosos including Stevie Wishart, a medieval fiddle-player and hurdy-gurdy enthusiast, and the Orlando Consort, a polyphonic vocal ensemble. Perhaps most significantly, however, the weekend will mark the London debut of her own newly formed group, the Division Lobby, a band that will devote itself to exploring the outer limits of 17th-century Italian improvisation, “a hot-bed of activity at the time”.

Chateauneuf has spoken of improvisation, provocatively, as “the final frontier” of the early music revival. Given the wave of interest in historically informed performance of Renaissance and baroque music over the past 30 years, it is surprising that it has not been more widely practised. Has the freedom to improvise actually been stifled by this voguish obsession with authenticity? “The early music world is amazingly subdued. There’s not a lot of improvisation within concerts and there should be more,” she says. “When you read about the kinds of things that went on, we’re hardly scratching the surface.”

There are already a handful of early music ensembles that improvise – viol quartet Phantasm, for example, and the jamming artists of L’Arpeggiata – but, as Chateauneuf explains, the fashion has been to draft in jazz or folk musicians “to do the hard bits”, resulting in a strange, if affecting, fusion of styles. The Division Lobby is more scholarly in its approach: its forthcoming programmes of sacred and secular works are the result of much truffle-hunting in libraries and careful research into performance practice. Many 17th-century concerts were breathtakingly ambitious: a French viol player called André Maugars described a spectacle at a church in Rome involving 10 choirs, each with its own organ and instrumentalists, and the exhilarating, antiphonic pieces that emerged, unrehearsed, as if through divine inspiration.

There will be nothing on quite that scale at Take The Risk, but the Division Lobby will use skeletal scores and work up simple chord sequences or a gently roving bass line into a rich tapestry of sound. Perhaps the biggest misconception about improvisation – and the measure of a fine performance – is that it’s easy to pull off: within this repertoire each split-second decision must be sensitive to the taste and style of the period, and informed by a deep understanding of the harmonic language that is dictated by a number of contemporary treatises.

In order for ideas to flourish, practice sessions must be intense, recurrent and, above all, democratic. It seems more than coincidental, therefore, that the demise of improvised music coincided with the emergence of the autocratic conductor, but there were other contributory factors, not least the commercialisation of music production. As the industry grew, so rehearsal times got tighter and programmes more prescribed, and business minds became keener to promote music that was as self-sufficient as possible. And with the technological developments of the 20th century came new inhibitions: record labels have long favoured the definitive over the experimental – audio airbrushing, if you like – so that the chance of grating mistakes is reduced, and with it the chance of inspired invention.

There does, however, appear to have been a backlash. Last year Mark Padmore broke convention with a conductorless performance of Bach’s St John Passion, Chandos has recently released a conductorless recording of Dido and Aeneas and keyboardists are becoming more daring in their interpretation of baroque works. Singers, too, are displaying more confidence and creativity: Renée Fleming, among others, has admitted that her interest in jazz has encouraged a looser and more fluid approach to Handel’s music, and there are many singers who will now happily ad-lib ornamentation on the night.

Performers often report better results with the added pressure of a live audience. “There’s that moment before you start when you don’t know what’s going to happen, and it’s frightening in a way but also very exciting, because if you put yourself in that position the creative muse tends to take over,” Chateauneuf says. Indeed, some argue that improvisation cannot truly exist without an audience to receive and respond. “It’s a very different experience as a musician to present an improvised piece. The whole atmosphere completely changes and I think it draws people in, in a way that’s quite different from composed music,” says Chateauneuf.

She admits that the Division Lobby’s future will depend on the public response to Take The Risk, but external trends point to a positive outcome. On the face of it baroque improvisation might seem a rather antiquated art but the practice is also remarkably in tune with the zeitgeist. In recent years there has been a growing demand for the inimitability of live performance, which is reflected not just in the world of classical music but in the pop industry too. Festivals are thriving and ticket prices to live gigs are soaring, while CD sales continue to plummet. The coolly sanitised sound world of the 21st century, where every note is perfected and recorded for us to play, pause and repeat ad infinitum, has left us with a strange sense of malaise, and value is now shifting towards the experience that offers spontaneity, a feeling of communion and the reassurance of uncertainty.

Take the Risk, Southbank Centre, London SE1, October 2-4,
www.southbankcentre.co.uk