Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search
The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.
User-agent: *
Disallow: /
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /wp-content/themes
Disallow: /wp-content/plugins
Disallow: /wp-content/upgrade
Disallow: /wp-content/themes_backup
Disallow: /wp-content/cache
Disallow: /xmlrpc.php
Disallow: /template.html
Disallow: /wp-comments
Disallow: /cgi-bin
Disallow: /trackback
Disallow: /feed
Disallow: /comments
Disallow: /comment-page
Disallow: /replytocom=
Disallow: /author
Disallow: /?author=
Disallow: /tag
Disallow: /?feed=
Disallow: /?s=
Disallow: /?se=
Host: edipresse.pl
