View Paper

A REVIEW OF WEB CRAWLERS FOR INFORMATION RETRIEVAL

Babatunde Ajose-Ismail & Ayodeji Osanyin, July 2020, Published: 2020-11-12

Performance of any search engine relies heavily on its Web crawler. Web crawlers are the programs that get webpages from the web by following hyperlinks. These webpages are indexed by a search engine and can be retrieved by a user query. In the area of web crawling which is a subfield of Information Retrieval, we still lack an exhaustive study that covers all crawling techniques. This study follows the guidelines of systematic literature review and applies it to the field of Web crawling. Existing literature about the web crawler is classified into different key subareas. Each subarea is further divided according to the techniques being used. We have highlighted future areas of research. We call for an increased awareness in various fields of the web crawler.
Download Brochure