Search engines flagging behind Web boom

Internet search engines are struggling to keep up with the continued expansion of the Web, according to two professors from the NEC Research Institute.

Research shows that the top 11 engines are actually getting worse at looking through the 800 million pages now on the Web. The last time performance was measured in December 1997, the main players reached 60 per cent of all pages this time the top 11 managed only 42 per cent.

One of the report's co-authors, Dr C Lee Giles, said: "We were surprised at the results. I think search engines are not trying to search the whole Web. Users want quality information rather than quantity."

He added: "Search engines need to be more specialised and more niche." Giles also complained that the interfaces treated users like "computer scientists" and called for interfaces to become more user friendly.

The survey looked at 11 major engines - AltaVista, EuroSeek, Excite, Google, HotBot, Infoseek, Lycos, Microsoft, Northern Light, Snap and Yahoo!. Northern Light came out on top with an estimated 16 per cent coverage.

The survey also found that only 34.2 per cent of sites used Meta tags - the label used to describe site content. It also discovered 123 distinct types of Meta tags suggesting a failure of standards.