Metacrawler

A metacrawler is a description of a search engine that utilizes a bot to crawl through websites and index the page by the data contained in the meta tags or the first portions of text on the page first developed by Erik Selberg. Because of abuse, major search engines have abandoned this technique as the only way to index web pages and now rely mainly on the content contained on the page.

Also see: Bot, Crawl, Index, Internet terms, Meta tags, Search engine