INDEXING & CRAWLING · FIELD GUIDE
Robots.txt blocking Google: check the exact rule
Fetch robots.txt from the affected URL’s own origin, identify the rule that matches Googlebot and check the exact path. Change only an accidental restriction. Robots.txt controls crawling; it is neither authentication nor a dependable way to remove an indexed URL.
Check the right host and deployed file
The apex domain and a subdomain can serve different robots files. Inspect the public response, not just a local file in your repository. Record status and content; a deployment, reverse proxy or generated environment flag may be serving a different rule than expected.
Illustrative case: the public site serves “User-agent: *” followed by “Disallow: /” after a staging configuration is deployed. That is a sitewide crawl block. Opening a page in your own browser does not test whether a crawler is permitted to request it.
Trace the matching path instead of deleting the file
Find the applicable user-agent group and the allow/disallow rules for the affected URL. A narrowly scoped restriction may be intentional. Test representative sibling URLs and resources before changing a broad prefix, especially when public documentation and account areas share part of a path.
- Record the affected hostname and full path.
- Check the matching user-agent group and path rule.
- Keep private areas protected by authentication regardless of robots rules.
- Verify that required scripts, styles and images are crawlable.
Do not combine blocking with an unreadable noindex
A crawler must fetch a page to see its noindex directive. Blocking that fetch can prevent the directive from being processed. Conversely, allowing crawling does not override a noindex in the response. Diagnose these as separate controls and choose the one that matches the intended outcome.
Deploy a narrow correction and inspect it again
Change the source that actually generates the public robots response. Preserve intentional restrictions and verify the deployed file plus a small set of affected URLs. Check HTTP status and indexing directives too; removing one robots block does not fix a response error or an accidental canonical.
Interpret monitoring failures honestly
A technical monitor may use a different user-agent from Googlebot. “Blocked for this monitor” is evidence about that request, not proof that Google was blocked. Save the exact rule and affected agent. After correction, separate a successful monitor fetch from Google’s later recrawl and indexing state.
Continue the investigation
- Page not indexed by Google: find the blocking stage
- Canonical and noindex: different signals, different checks
- Sitemap errors: separate fetching from URL quality
Primary references
These references describe Google’s reporting and crawling behavior. The worked examples above are illustrative, not customer results.