Check your robots.txt file
Check whether a website has a robots.txt file, review its rules and see any sitemap declarations it contains. Free and read-only.
Free and read-only. Solve only requests public web resources and does not change the website.