# robots.txt for http://www-ai.cs.uni-dortmund.de # # comments to: webadmin@ls8.cs.uni-dortmund.de # info about the syntax: # http://info.webcrawler.com/mak/projects/robots/norobots-rfc.html # So far, every agent can read everything, expect of our internal # pages, which are protected anyhow. User-agent: CiteSeerBot Disallow: /INTERN Disallow: /SYSTEM User-agent: * Disallow: /INTERN Disallow: /SYSTEM