Indexing FOR WEB SEARCH

Indexing for Web Search



Indexing for Web Search

Introduction

The introduction and growth of the World Wide Web (WWW, or Web) have resulted in a profound change in the way individuals and organizations access information. In terms of volume, nature, and accessibility, the characteristics of electronic information are significantly different from those of even five or six years ago. Control of, and access to, this flood of information rely heavily on automated techniques for indexing and retrieval. According to Gudivada, Raghavan, Grosky, and Kasanagottu, “The ability to search and retrieve information from the Web efficiently and effectively is an enabling technology for realizing its full potential.” Almost 93 percent of those surveyed consider the Web an “indispensable” Internet technology, second only to e-mail. Although there are other ways of locating information on the Web (browsing or following directory structures), 85 percent of users identify Web pages by means of a search engine (Bharat, 2008, 20).

A more recent study conducted by the Stanford Institute for the Quantitative Study of Society confirms the finding that Searching for information is second only to e-mail as an Internet activity. In fact, Nie and Ebring conclude, “… the Internet today is a giant public library with a decidedly commercial tilt. The most widespread use of the Internet today is as an information search utility for products, travel, hobbies, and general information. Virtually all users interviewed responded that they engaged in one or more of these information gathering activities.”

Techniques for automated indexing and information retrieval (IR) have been developed, tested, and refined over the past 40 years, and are well documented. With the introduction of the Web, and the capability to index and retrieve via search engines, these techniques have been extended to a new environment. They have been adopted, altered, and in some cases extended to include new methods. “In short, search engines are indispensable for searching the Web, they employ a variety of relatively advanced IR techniques, and there are some peculiar aspects of search engines that make searching the Web different than more conventional information retrieval”.

The environment for information retrieval on the World Wide Web differs from that of “conventional” information retrieval in a number of fundamental ways. The collection is very large and changes continuously, with pages being added, deleted, and altered. Wide variability between the size, structure, focus, quality, and usefulness of documents makes Web documents much more heterogeneous than a typical electronic document collection. The wide variety of document types includes images, video, audio, and scripts, as well as many different document languages. Duplication of documents and sites is common. Documents are interconnected through networks of hyperlinks. Because of the size and dynamic nature of the Web, preprocessing all documents requires considerable resources and is often not feasible, certainly not on the frequent basis required to ensure currency. Query length is usually much shorter than in other environments—only a few words—and user behavior differs from that in other environments (Baeza, 2009, ...