Crawl

Crawl is a term used to describe the process of a bot, script, or other software program that visits a web page and grabs the contents and links in that page. Once completed the program visits the next link in the list or one of links obtained from the web page it just visited.