Find sitemap.xml, robots.txt, ads.txt, app-ads.txt and sellers.json - Real-Time Search
The robots.txt file is a file at the root a domain indicates parts of the site should not be accessed by search engine crawlers.
# Robots file for Calciomercato.com
User-agent: *
Disallow: /*?c=
Disallow: /*?a=
Disallow: /*?n=
Disallow: /*?page=
Disallow: /*?reply=
Disallow: /siteComments/voteUp/
Disallow: /siteComments/voteDown/
Disallow: /eyeblaster
Disallow: /addineyeV2.html
Disallow: /comments
Disallow: /news-related
Sitemap: https://www.calciomercato.com/sitemap-news.xml
Sitemap: https://www.calciomercato.com/sitemap-news-en.xml
User-agent: Mediapartners-Google
Disallow: /gallery/
