Site Search

Find sitemap.xml, robots.txt and ads.txt - Real-Time Search

Not http https www subdomain robots.txt

The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.

(*) Thumbnail Screenshots by ShrinkTheWeb

User-agent: *
Disallow: /cgi-bin/aws_tool*
Disallow: /cgi-bin/itemlist*
Disallow: /cgi-bin/find*

User-Agent: BaiduMobaide
Disallow: /

User-Agent: Baiduspider
Disallow: /

User-Agent: Baiduspider/2.0
Disallow: /

user-agent: Baiduspider+
Disallow: /

user-agent: BaiduImagespider
Disallow: /

User-agent: yodaobot
Disallow: /

User-agent: Yeti
Disallow: /

User-agent: NaverBot
Disallow: /

User-agent: Slurp
Crawl-delay: 2

User-agent: msnbot
Crawl-delay: 120

Tuesday, 21 May 2019