.Google's John Mueller addressed a question regarding why Google marks pages that are actually prohibited coming from creeping through robots.txt as well as why the it's risk-free to disregard the relevant Search Console records about those crawls.Robot Web Traffic To Inquiry Criterion URLs.The individual talking to the concern documented that robots were creating web links to non-existent concern parameter Links (? q= xyz) to web pages along with noindex meta tags that are actually likewise shut out in robots.txt. What caused the question is actually that Google.com is creeping the hyperlinks to those web pages, obtaining obstructed by robots.txt (without envisioning a noindex robotics meta tag) after that obtaining reported in Google.com Look Console as "Indexed, though blocked through robots.txt.".The individual inquired the adhering to concern:." However here's the major concern: why would certainly Google.com index webpages when they can't even see the material? What's the advantage during that?".Google.com's John Mueller validated that if they can't creep the web page they can't observe the noindex meta tag. He also creates an interesting reference of the internet site: search driver, urging to ignore the results because the "average" users won't find those end results.He composed:." Yes, you're proper: if our experts can not crawl the webpage, our experts can not see the noindex. That stated, if our experts can not creep the pages, after that there is actually not a lot for us to index. Thus while you could view a number of those pages along with a targeted internet site:- concern, the average consumer won't observe them, so I would not fuss over it. Noindex is also great (without robots.txt disallow), it merely implies the URLs will certainly end up being actually crept (and also end up in the Look Console record for crawled/not listed-- neither of these conditions induce problems to the rest of the internet site). The vital part is that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's solution verifies the constraints in using the Internet site: hunt evolved search driver for analysis explanations. Among those factors is actually considering that it's not linked to the regular search index, it is actually a separate trait completely.Google.com's John Mueller talked about the web site hunt driver in 2021:." The quick response is actually that a web site: query is actually certainly not indicated to become total, neither utilized for diagnostics objectives.A site query is actually a specific type of search that limits the outcomes to a specific website. It is actually generally just words website, a colon, and afterwards the web site's domain name.This query restricts the end results to a details site. It's not suggested to be a detailed collection of all the web pages from that site.".2. Noindex tag without utilizing a robots.txt is actually great for these type of situations where a bot is linking to non-existent pages that are receiving uncovered by Googlebot.3. Links with the noindex tag are going to produce a "crawled/not listed" entry in Browse Console and that those will not have a negative impact on the remainder of the internet site.Check out the concern and also address on LinkedIn:.Why would certainly Google mark pages when they can not even see the material?Featured Picture through Shutterstock/Krakenimages. com.