Site Search

Find sitemap.xml, robots.txt and ads.txt - Real-Time Search

Not http https www subdomain robots.txt

The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.

(*) Thumbnail Screenshots by ShrinkTheWeb

User-agent: Google
Craw-delay: 0

User-agent: Yahoo
Crawl-delay: 0

User-agent: Bing
Craw-delay: 0

User-agent: *
Crawl-delay: 10

Saturday, 24 August 2019