Developer Robots.txt generator
Build a robots.txt with multiple user-agent groups, sitemap declarations and an optional block for AI training crawlers.
Group 1
Seconds. Google ignores this; Bing and Yandex honour it.
One path per line.
Exceptions inside a disallowed folder.
One per line, absolute.
Adds GPTBot, ClaudeBot, Google-Extended, CCBot and anthropic-ai. These are honoured voluntarily. Robots.txt is a request, not an access control.
robots.txt
7 linesUser-agent: * Disallow: /admin/ Disallow: /cart/ Disallow: /*?s= Sitemap: https://example.com/sitemap.xml
Disallow is not noindex. Blocking a URL in robots.txt stops it being crawled, but it can still be indexed from external links, and because the crawler cannot fetch the page, it will never see a noindex tag on it. To keep a page out of the index, allow crawling and use <meta name="robots" content="noindex">. To keep it private, require authentication.
Ready. Runs locally on your device.
Your files stay on your device. The tool works directly in your browser, using your device to process your files. Nothing is sent to our servers, and we never receive, store, or see your files or figures.
Share result
Cite
Cite this page
Reembun. (2026, July 28). Robots.txt generator. https://reembun.com/robots-txt-generator
How to use it
Choose the crawler
User-agent names who each block applies to. A star covers every crawler that has no block of its own.
List the paths
One path per line under Disallow. Allow carves out the exceptions inside a folder you have blocked.
Add your sitemaps
Absolute URLs, one per line. This is the line that tells a crawler where the map of your site lives.
Copy and publish
Add a block for AI training crawlers if you want them out. Save the result as robots.txt in the root of the domain, because nowhere else is read.
The format
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/public/
Sitemap: https://example.com/sitemap.xmlRules are grouped by User-agent. A crawler obeys the most specific group that names it, and ignores the rest, so a bot named explicitly will not also apply the * rules.
| Directive | Effect |
|---|---|
User-agent | Which crawler the following rules apply to |
Disallow | Path prefix not to crawl |
Allow | Exception inside a disallowed path |
Sitemap | Absolute URL of a sitemap; applies to all crawlers |
Crawl-delay | Seconds between requests; ignored by Google |
An empty Disallow: means allow everything, which is clearer written out than omitted.
Disallow is not noindex
This is the mistake that removes pages from search by accident, and the one that fails to.
Disallow stops a crawler fetching the URL. The URL can still be indexed from external links, typically appearing with no description and a note that the page could not be crawled.
Worse, blocking a page means the crawler never sees a noindex tag on it. Adding both is self-defeating: the block prevents the instruction being read.
| Goal | Method |
|---|---|
| Keep out of the index | Allow crawling, add <meta name="robots" content="noindex"> |
| Save crawl budget on low-value URLs | Disallow in robots.txt |
| Keep genuinely private | Authentication; robots.txt is public |
That last row deserves emphasis. robots.txt is world-readable at a predictable path, so listing a sensitive directory in it advertises the directory’s existence to anyone who looks.
Wildcards
Google and Bing support two non-standard extensions:
Disallow: /*? # any URL with a query string
Disallow: /*.pdf$ # any URL ending in .pdf* matches any sequence and $ anchors the end. Not all crawlers support them, so do not rely on them for anything critical.
AI training crawlers
A distinct set of user agents, all of which publish that they honour robots.txt:
| Agent | Operator | Purpose |
|---|---|---|
GPTBot | OpenAI | Model training |
ClaudeBot | Anthropic | Model training |
Google-Extended | Gemini training and grounding | |
CCBot | Common Crawl | Public crawl corpus |
PerplexityBot | Perplexity | Search and answers |
Blocking these does not affect search ranking, since Google-Extended is specifically separate from Googlebot for that reason.
Whether to block is a business decision rather than a technical one. Blocking protects content from training use; allowing may mean your content is cited in AI answers, which is increasingly a referral channel. There is no consensus, and the honest position is that the trade-off is still unsettled.
Placement and testing
The file must live at the root: https://example.com/robots.txt. A file at /blog/robots.txt is ignored entirely. Each subdomain needs its own.
Test with Google Search Console’s robots.txt report before and after publishing. A stray Disallow: / has removed entire sites from search, and it usually happens when a staging configuration is deployed to production.
Common questions
Does robots.txt keep a page out of Google?
No, and this is the most consequential misunderstanding about it. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other sites link to it, and because the crawler cannot fetch the page, it will never see a noindex tag you put there. To deindex, allow crawling and use a noindex meta tag.
Does blocking AI crawlers actually work?
Only for those that choose to honour it. GPTBot, ClaudeBot, Google-Extended and CCBot all publish that they respect robots.txt. Nothing enforces it. Robots.txt is a request, not access control. Anything that must not be read needs authentication.
What is Google-Extended?
A token that controls whether your content is used for Gemini training and grounding, without affecting Google Search crawling or ranking. Blocking Googlebot would remove you from search; blocking Google-Extended does not.
Does crawl-delay work?
Google ignores it entirely and uses crawl rate settings in Search Console instead. Bing, Yandex and several smaller crawlers do honour it. If Googlebot is overloading your server, the fix is server-side rate limiting or a Search Console adjustment.
Last reviewed
