Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search
The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.
User-agent: *
Disallow: /wp-admin/
Disallow: /avb/
Disallow: /wp-content/uploads/2017/05/Allgemeine-Vertragsbedingungen-TRG.pdf
Disallow: /datenschutzerklaerung-fuer-popunder/
Allow: /wp-admin/admin-ajax.php
