In an oversimplification, Robots. txt are text files that Google’s search engine spiders read, and that adhere to a strict format. Because it’s a direct line of conversation to the “spiders” which are enticing along with your web page, only one error can negatively impact all of your platform. According to Yoast, “Search engines index the internet by spidering pages, following links to go from site A to site B to site C and so on. Before a search engine spiders any page on a domain it hasn’t encountered before, it’ll open that domain’s robots.
txt file, which tells the search engine which URLs on that site it’s allowed to index.