# /robots.txt file for http://webcrawler.com/ # mail webmaster@webcrawler.com for constructive criticism User-agent: Slurp Disallow: /Tournaments User-agent: Slurp Crawl-delay: 50 User-agent: msnbot Disallow: /Tournaments User-agent: msnbot Crawl-delay: 50