.Google's John Mueller responded to a question about why Google marks web pages that are forbidden from creeping by robots.txt and also why the it's safe to overlook the similar Look Console records regarding those creeps.Robot Visitor Traffic To Concern Specification URLs.The person talking to the inquiry documented that crawlers were making links to non-existent query guideline Links (? q= xyz) to webpages with noindex meta tags that are actually additionally obstructed in robots.txt. What cued the inquiry is actually that Google.com is creeping the links to those pages, acquiring blocked through robots.txt (without envisioning a noindex robotics meta tag) after that obtaining turned up in Google Browse Console as "Indexed, though blocked out through robots.txt.".The person talked to the adhering to concern:." Yet listed below's the big inquiry: why would certainly Google mark pages when they can not also view the information? What is actually the conveniences during that?".Google's John Mueller verified that if they can't creep the page they can not view the noindex meta tag. He also creates a fascinating reference of the web site: search operator, suggesting to overlook the results given that the "normal" users won't find those end results.He created:." Yes, you are actually proper: if we can't creep the webpage, we can not see the noindex. That mentioned, if our company can not crawl the webpages, then there is actually not a whole lot for our team to index. Thus while you might see a number of those pages along with a targeted web site:- question, the average customer will not view them, so I wouldn't bother it. Noindex is also alright (without robots.txt disallow), it merely means the Links will certainly wind up being actually crept (and also end up in the Search Console file for crawled/not recorded-- neither of these standings lead to issues to the rest of the internet site). The important part is that you don't make all of them crawlable + indexable.".Takeaways:.1. Mueller's response confirms the constraints in using the Internet site: search progressed hunt driver for analysis explanations. One of those reasons is actually since it's not linked to the frequent search index, it is actually a distinct point entirely.Google's John Mueller talked about the internet site search operator in 2021:." The brief solution is actually that an internet site: inquiry is not suggested to become complete, nor made use of for diagnostics purposes.An internet site question is a certain sort of hunt that limits the outcomes to a particular internet site. It is actually basically only the word site, a digestive tract, and afterwards the site's domain name.This question limits the results to a particular website. It's not indicated to become a comprehensive assortment of all the web pages coming from that website.".2. Noindex tag without using a robots.txt is actually great for these sort of scenarios where a crawler is actually connecting to non-existent web pages that are acquiring uncovered by Googlebot.3. URLs with the noindex tag will generate a "crawled/not recorded" item in Browse Console and that those will not have a bad effect on the rest of the site.Go through the concern and address on LinkedIn:.Why would Google.com mark web pages when they can not even view the content?Featured Photo through Shutterstock/Krakenimages. com.