For More Information
For more information on web clients, refer to:
http://www.robotstxt.org/wc/robots.html
The Web Robots Pages-resources for robot developers, including the registry of Internet Robots.
http://www.searchengineworld.com
Search Engine World-resources for search engines and robots.
Search Tools for Web Sites and Intranets-resources for search tools and robots.
http://search.cpan.org/doc/ILYAZ/perl_ste/WWW/RobotRules.pm
RobotRules Perl source.
http://www.conman.org/people/spc/robots2.html
An Extended Standard for Robot Exclusion.
Managing Gigabytes: Compressing and Indexing Documents and Images
Witten, I., Moffat, A., and Bell, T., Morgan Kaufmann.