Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search
The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.
User-agent: *
Disallow: /cgi-bin
Disallow: /?
Disallow: /wp-
Disallow: /wp/
Disallow: *?s=
Disallow: *&s=
Disallow: /search/
Disallow: /author/
Disallow: /users/
Disallow: */trackback
Disallow: */feed
Disallow: */rss
Disallow: */embed
Disallow: */wlwmanifest.xml
Disallow: /xmlrpc.php
Disallow: *utm*=
Disallow: *openstat=
Disallow: *.pdf
Disallow: *.docx
Disallow: *.pptx
Allow: */uploads
Allow: */*.css
Allow: */*.js
Allow: */*.gif
Allow: */*.svg
Allow: */*.png
Allow: */*.jpg
Allow: */*.jpeg
Sitemap: https://admixer.com/sitemap_index.xml
Sitemap: https://creatives.admixer.com/sitemap.xml
Sitemap: https://blog.admixer.com/sitemap_index.xml
Host: admixer.com
