# ROBOTS.TXT - STOP WEBCRAWLERS / SPIDERS FROM # FILLING UP LOG FILES WITH IRRELEVANT SEARCHES User-agent: * Disallow: /history/courses/ # NO LONGER USED BY HISTORY Disallow: /navigation/ # DOES NOT EXIST Disallow: /local/ # internal service tools