Where the file goes
robots.txt must sit at the root of the domain — https://example.com/robots.txt. A file in a subfolder is ignored entirely. Each subdomain needs its own, and http and https are treated as separate hosts by some crawlers, so serve it on both if you have not redirected one to the other.
The rule everyone gets wrong
Disallow stops a crawler from fetching a page. It does not remove the page from search results. If other sites link to a blocked URL, Google can still list it — with no description, because it was never allowed to read the page. To actually keep something out of the index you must let it be crawled and serve a noindex meta tag or header. Blocking a page you have also marked noindex is the classic mistake: the crawler can never see the noindex, so the page stays listed.
Never use it to hide anything sensitive
robots.txt is a public file, and listing /admin-backup/ in it is a signpost, not a lock. Anyone can read yours right now. Well-behaved crawlers obey it; scrapers and attackers read it for ideas. Protect private areas with authentication.
About the AI crawler options
These user-agents are how the major AI companies say they identify themselves, and blocking them is the standard way to opt out of having your content used for training or retrieval. Two honest caveats. First, compliance is voluntary — a directive is a request, not enforcement, and crawlers that ignore robots.txt will ignore this too. Second, some of these agents also power features that send you traffic, so blocking everything can cost you referrals. Blocking training crawlers while allowing search crawlers is the balance most publishers choose.
Wildcards and testing
Google and Bing support * as a wildcard and $ to anchor the end of a URL, so Disallow: /*.pdf$ blocks PDFs. Not every crawler supports either. After publishing, test the file in Google Search Console's robots.txt report — a single stray character in the wrong place can block your entire site, and it is a genuinely common way to lose all your traffic overnight.
Benefits
- Produces a valid robots.txt with crawl rules, allow exceptions and a sitemap directive.
- One-tick blocking for the named AI training crawlers, which most generators still omit.
- Presets for WordPress and online stores cover the paths those platforms usually need blocked.
- Warns when your settings would block the whole site from search engines.
Limitations to know
- robots.txt controls crawling, not indexing — a blocked URL can still be listed if other sites link to it.
- Compliance is voluntary; scrapers and malicious bots simply ignore the file.
- Wildcard and $ anchor support varies between crawlers, so complex patterns are not universally honoured.
Common mistakes to avoid
- Blocking a page in robots.txt and also adding a noindex tag — the crawler can never read the tag, so the page stays indexed.
- Listing private folders, which turns a public file into a map of what you would rather hide.
- Leaving a staging site’s "Disallow: /" in place after launch, which quietly removes the live site from search.
Alternatives
To add a proper noindex tag instead of a crawl block, use the Meta Tag Generator. To check how pages look when shared, use the Open Graph Preview.
Last updated: August 2026 · Reviewed by the AI Toolbox editorial team.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No. It only stops crawling. Use a noindex meta tag on a crawlable page to remove it from the index.
Do AI crawlers have to obey these rules?
Compliance is voluntary. The major AI companies publish these user-agent names and state that they honour robots.txt, but nothing technically enforces it.
Should I block Google-Extended?
It controls use of your content for Gemini training only — it does not affect your Google Search ranking or crawling. Blocking it is a content-licensing decision, not an SEO one.
What happens if I have no robots.txt at all?
Crawlers assume everything is allowed. That is fine for most sites; a file is only needed when you want to restrict something or point at your sitemap.