Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search
The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.
User-agent: *
Disallow: /request/
Disallow: /*/request/
Disallow: /drafts/
Disallow: /*/drafts/
Disallow: /rss_items/
Disallow: /*/rss_items/
Disallow: /redirects/
Disallow: /*/redirects/
Disallow: /cmr/
Disallow: /*/cmr/
Disallow: /layout/set/popup/
Disallow: /*/layout/set/popup/
Disallow: /*.pdf$
Disallow: /rus$
Disallow: /dut$
Disallow: /rus/*
Disallow: /dut/*
Disallow: /Request/
Disallow: /*/Request/
Disallow: /Drafts/
Disallow: /*/Drafts/
Disallow: /Rss_items/
Disallow: /*/Rss_items/
Disallow: /Redirects/
Disallow: /*/Redirects/
Disallow: /Cmr/
Disallow: /*/Cmr/
Disallow: /Layout/set/popup/
Disallow: /*/Layout/set/popup/
Disallow: /*.Pdf$
Disallow: /Rus$
Disallow: /Dut$
Disallow: /Rus/*
Disallow: /Dut/*
User-agent: AhrefsBot
Disallow: /
