User-Agent: OmniExplorer_Bot Disallow: / User-agent: * Disallow: /api-doc/ Disallow: /acs-admin/ Disallow: /acs-developer-support/ Disallow: /acs-service-contract/ Disallow: /cms/ Disallow: /doc/ Disallow: /pvt/ Disallow: /sa/ Disallow: /scp/ Crawl-delay: 1 # min of 1 second between requests Request-rate: 3/7 # max of 3 page requests in 7 seconds Visit-time: 1913-2333 # Restrict indexing to 19:13-23:33 GMT (~1-5 EST, ~4-8 PDT) ################################################################################ # NOTE: more complex rules are placed last so simpler robots will at least see # the most important parts. See http://www.conman.org/people/spc/robots2.html