Search Engine Web Crawlers

They go by a variety of names - bots, worms, web spiders, automatic indexers, web robots and web crawlers. What they are really are programs or automated scripts that browse the web in a methodical and automated manner. They are known best as web crawlers because that is what they do – 'crawl' a website in order to know what it contains.

Read The Full Article Here:
http://www.ekonline.com/web_design_articles/crawlers.html