Site Search

Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search

Not http https www subdomain robots.txt

The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.

(*) Thumbnail Screenshots by ShrinkTheWeb

# 1. A robot may not injure a human being or, through inaction, allow a
# human being to come to harm.
# 2. A robot must obey orders given it by human beings except where such
# orders would conflict with the First Law.
# 3. A robot must protect its own existence as long as such protection
# does not conflict with the First or Second Law.

User-agent: *
Disallow: /tag 

Disallow: /cat 

Friday, 29 May 2020