For more information on web clients, refer to:

http://www.robotstxt.org/wc/robots.html

The Web Robots Pages-resources for robot developers, including the registry of Internet Robots.

http://www.searchengineworld.com

Search Engine World-resources for search engines and robots.

http://www.searchtools.com

Search Tools for Web Sites and Intranets-resources for search tools and robots.

http://search.cpan.org/doc/ILYAZ/perl_ste/WWW/RobotRules.pm

RobotRules Perl source.

http://www.conman.org/people/spc/robots2.html

An Extended Standard for Robot Exclusion.

Managing Gigabytes: Compressing and Indexing Documents and Images

Witten, I., Moffat, A., and Bell, T., Morgan Kaufmann.

 


Hypertext Transfer Protocol (HTTP)