Robots.txt

`robots.txt` at the site root gives **crawl guidance** to well-behaved bots via `User-agent`, `Disallow`, `Allow`, and `Sitemap` directives. It is **not** access control—private URLs remain discoverable if linked elsewhere.

					User-agent: *
Disallow: /admin/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xml
				

| Directive | Effect | |-----------|--------| | Disallow | Ask not to crawl matching paths | | Allow | Exceptions within disallowed trees | | Sitemap | Points to XML sitemap location |

Use `meta name="robots"` or `X-Robots-Tag` for page-level index/noindex; robots.txt cannot noindex a page that is still crawled if linked.

On interviews: robots.txt vs authentication; blocking staging; unintended block of `/static/` assets; Googlebot smartphone vs desktop.

Common pitfalls: blocking CSS/JS needed for rendering; using Disallow for secrets; conflicting robots meta and txt; forgetting robots.txt on new subdomain.

The trade-off is limiting crawler noise versus accidentally hiding resources needed for rendering.

Checklist:

• Never rely on robots.txt for security.
• Do not block render-critical assets.
• Document staging vs production rules.
• Test with Search Console URL inspection.