Financial Times FT.com

Engine of the superhighway

By Peter Martin

Published: December 29 1995 11:23 | Last updated: August 22 2002 12:23

Every medium invents its own distinctive format, the one which best takes advantage of its unique characteristics. For moving pictures, it was the big screen action drama. For radio, the quiz show. For television, the situation comedy and - more recently - the confessional chat show.

Usually these happen soon after the medium has come into existence, often while it is still technically primitive. The initial version may require a great deal of indulgence on the part of its consumers, just as silent pictures did before the first world war, or radio did in the 1920s. But it stamps itself - and the medium in which it appears - indelibly on the public mind.

The innovative new format may not be the only one each medium employs; it may not even be the most important, commercially speaking. But these new formats are important because they reflect the most significant aspects of the medium, capturing its appeal. Most important, they provide a sign that it is coming of age in its own right.

The World Wide Web, the fastest-growing part of the Internet, has been widely accessible for two years. But it has already evolved its own distinct format, in which we can discern how the Web will evolve.

The format in question goes by the unglamorous name of the “search engine”. Individual examples are even more bafflingly named: , Lycos, Inktomi, Webcrawler, and a score of others.

Before examining how the search engine fits its new medium, let us spend a moment on the Web itself. It was devised by Tim Berners-Lee, a scientist at CERN, the European centre for research in nuclear physics, to allow researchers to navigate quickly and easily around widely scattered databases of academic papers and experimental studies.

In its initial form it consisted of plain type on screen, but allowed the creation of “hypertext links”, so a user in Asia using a computer to read a paper held on a US computer could look up a passing reference to another study held on a computer in Europe, and immediately switch to its full text. From there, the user could switch to other references.

The process of switching from subject to subject was like channel- surfing on television, hence the phrase “surfing the Net”. The comparison became even more apt when Marc Andreessen, a computer scientist, invented Mosaic, a graphical front end to the Web.

Mosaic and its successors are “browser” programs, which run on a user’s computer to simplify surfing. They also make the presentation of the results more visually appealing: full-colour diagrams, pictures, drawings and photographs can be integrated into the text. Users can click on highlighted words or pictures to switch from computer to computer and subject to subject around the world. As competing browsers have appeared - the best-known is Andreessen’s new Netscape Navigator - they have become ever more powerful, adding the capacity to download animation, sound and even video, though most websites restrict themselves to words and still pictures.

Websites have proliferated at amazing speed, since the language in which they must be written is well within the grasp of the average hobbyist. Some are extremely professional, while others reflect the idiosyncracies of their authors, such as All Men Must DIE, a website constructed by a young woman in London who is on the rebound from an unhappy relationship.

The sites themselves are not inherently original: they have borrowed heavily from magazines and television. Even a very elaborate site like HotWired, the electronic offshoot of San Francisco’s Wired magazine, relies on formats derived from avant-garde graphic design and music television.

This is where the search engines come in. They are sites you dial up with your browser that help you search through the other offerings on the Web. Each engine contains an index of documents, built up through continuous electronic patrolling of the Net. Such tools provide a powerful way of navigating the Web. Without these tools, the Web would lose much of its usefulness, since it is too large and complex for browsing or surfing.

One way of using a search engine is to work through increasingly specific tiers of menus to reach the subject category in which you are interested. But there is a more powerful way of searching. To use this, you type in the subject or name about which you are interested, and you get back a list of relevant locations, either websites or documents held in other parts of the Net.

Most search engines offer variants on this simple recipe. Some have more complex search routines, ranking sites by the probability that they will meet your needs - for example, a document in which your sought-for phrase appears in a headline will score more highly than one in which it is buried in body text.

Others have hand-selected indexes, winnowing out the irrelevant. Still others offer independent comment on the contents of a site or document. Typically, search engines try to cover their costs by selling advertising space.

The search engines may seem, on the face of it, to be no more than a convenience, but they illustrate the Web’s greatest strength and provide a solution to its greatest drawback.

The strength is the access, on an equal footing and with equal convenience, to contents provided by people around the world. The drawback is the lack of editing that this egalitarian structure involves: no quality control, no coherence, no selection.

One solution to this lack of coherence is the creation of self-contained but tightly edited “branded communities”. Publishers aim to suck Web users into a community, and to offer them almost all they need without going outside. Typically, these communities offer a range of channels or themes, ranging from sport to personal relationships and (inevitably) computers and the Web itself. Each channel offers original journalism, background material, contributions and comments from users, and sometimes downloadable software.

Such communities offer the possibility of subscription revenue, and of better opportunities to generate advertising revenue, since more is known about their members. Both Microsoft Network and Europe Online, which once saw themselves as proprietary networks competing with the Web, now see their future as branded communities within it.

In the long run, however, this approach is self-defeating. Because it uses the Web principally as a way to lure users into the cul-de-sac of the branded community, it surrenders the most attractive aspect of the medium, its breadth and heterogeneity of access.

Search engines provide a genuinely new publishing vehicle: a sort of interactive contents page that “edits” the contents of the Web, presenting it to the user in a way which is both more coherent and structured than mere browsing, but more heterogeneous and random than the branded communities.

The search engines have realised their power. Yahoo, which is probably the best known, is branching into publishing an ink-on-paper magazine. Several have added their own forms of information, such as a feed of wire-service news. Others will increasingly specialise in subject areas, style or censorship approach. , for example, does not list explicitly sexual sites. Others do.)

The future of Web publishing lies in the convergence of the approach offered by the search engines and that of the branded communities. Specialised search engines will add original material of their own and forums for public discussion, in the manner of the branded communities. The communities, in their turn, will add searching capabilities to bring the rest of the wider Web within users’ reach.

All this assumes that the bulk of the material on the Web will continue to be free or advertiser supported. For hobby sites, and those put up by companies promoting their wares, free access is the whole point of the exercise. For other sites, however, publishers will try to obtain revenues from users, just as they do for newspapers and magazines. Only very specialised information is likely to be able to justify a pay-per-view or subscription fee, however. Most Web publishers are likely to have to rely on advertising as their principal revenue source.

Web publishing in this way will partly replace ink-on-paper publishing, partly complement it, as the example shows. Technology will have to advance several steps further - in the ease of connecting to the Net, in the speed of transferring information, in the power of computers to cope with more complex animated illustrations - before it becomes a true mass medium. But just as anyone watching the Lumie’re Brothers’ short film L’Arroseur Arrose on its first showing 100 years ago this month could dimly discern the shape of the cinema of the future, so those using the Web now can see a new medium taking shape before their eyes.