December 3, 2006
Robots Overview
Robots are programs designed to travel through the web in order to find Websites to
store in databases for search engines to analyze and use.
Robots typically find their websites by visiting the sites already in their database and following the links
on those pages to new pages. When a Robot finds a link to a certain Website it notes what pages link to it
and performs calculation to assign that links points based on different Website variables. Most high end
search engines use their on algorythm to calculate a pages popularity. These calculations have several
hundred different factors including keyword density, links to the page, traffic to the page and how frequently
the page is updated. These calculations are done on a per page basis, as opposed to a per site basis.
Robots are also known as: Crawlers, WWW or Web Robots, Spiders, Web Wanderers, etc...
|