Robots.txt Generator
Create robots.txt files for search engine crawlers
How to Use Robots.txt Generator
- Choose which crawlers the rules apply to.
- Add the paths you want to disallow or explicitly allow.
- Add your sitemap URL.
- Copy the output and save it as robots.txt in your site root.
About Robots.txt Generator
What this page solves
A hand-written `robots.txt` with one misplaced `Disallow: /` can deindex an entire site, and generic generators emit rules that block the crawlers you actually want.
Robots.txt Generator builds a valid `robots.txt` from explicit choices — which crawlers to allow, which paths to block, and where your sitemap lives — with a live preview of the exact file, generated in your browser. The single most damaging mistake in SEO is shipping a staging `Disallow: /` to production; the generator makes that rule visible and deliberate instead of an inherited leftover, and warns when a rule would block your whole site. You can set rules per user-agent, so Googlebot and Bingbot get full access while `GPTBot`, `anthropic-ai`, `CCBot`, or `PerplexityBot` are allowed or blocked on your terms — an increasingly common requirement as AI crawlers grow. Remember what `robots.txt` does and does not do: it asks well-behaved crawlers not to fetch a path, but it does not keep a page out of the index if other sites link to it, and it does not protect anything private — use `noindex` and authentication for those. Pair it with the Canonical Tag Generator to consolidate duplicates, or the Meta Description Checker to audit the pages you are keeping crawlable. Runs entirely in your browser — no upload, no signup, no logs.
Common Use Cases
- Generate a production robots.txt at launch without inheriting staging rules
- Block admin, cart, and internal search paths from being crawled
- Allow or block AI crawlers such as GPTBot, CCBot, and PerplexityBot deliberately
- Add the sitemap directive so search engines discover your URL list
- Set different rules for Googlebot, Bingbot, and everything else
- Audit an existing robots.txt for a rule that is blocking pages by accident
Related tools in Developer Tools
- JSON FormatterFormat, beautify and validate JSON data
- JSON ViewerBrowse JSON as a collapsible tree with syntax colors
- JSON ParserParse JSON and inspect keys, arrays, depth and node count
- JSON Pretty PrintPretty print JSON with 2 or 4 space indent
See all Developer Tools.
Frequently asked questions
Does blocking a page in robots.txt remove it from Google?
No, and this is the most consequential misunderstanding about the file. Disallow prevents crawling. If other sites link to the page, Google can still index the URL — it just shows it without a description. To remove a page from the index, allow crawling and serve a noindex meta tag.
Where exactly does the file go?
At the root of the domain, served at /robots.txt. Crawlers look nowhere else. Rules in a robots.txt on a subdomain apply only to that subdomain.
Should I block my admin pages?
Be careful — robots.txt is public, so listing your admin paths tells everyone where they are. Protect them with authentication. If they are properly protected, there is little value in blocking them and some risk in advertising them.
Do all crawlers obey it?
The major search engines do. It is a voluntary convention with no enforcement, so scrapers and malicious bots routinely ignore it. Treat it as guidance for well-behaved crawlers, never as access control.
How to use Robots.txt Generator?
1) Open the tool. 2) Enter or upload your input. 3) Get your result instantly. Everything happens locally — nothing is uploaded.
When should I use Robots.txt Generator?
Use it when launching a site, moving from staging to production, or adding rules for AI crawlers and specific bots.
Is Robots.txt Generator safe?
Yes. Robots.txt Generator runs 100% in your browser. Inputs are never uploaded, stored, or logged — safe for confidential content.
Will quality be affected by Robots.txt Generator?
Robots.txt Generator performs lossless operations — your content is not re-encoded or downgraded.
What formats does Robots.txt Generator support?
Robots.txt Generator supports JSON, YAML, XML, CSV, TOML and other developer-friendly text formats
Is Robots.txt Generator free?
Yes — Robots.txt Generator is completely free, with no registration, no watermarks, and no usage limits.
Where does robots.txt have to live?
At the domain root — `https://example.com/robots.txt`. A file at any other path is ignored, and subdomains each need their own.
Does Disallow remove a page from Google?
No. It stops the page being crawled, but a URL linked from elsewhere can still appear in results without a snippet. To remove a page, allow crawling and serve a `noindex` tag — Google must fetch the page to see it.
How do I block AI crawlers?
Add a per-user-agent block for `GPTBot`, `anthropic-ai`, `CCBot`, `PerplexityBot`, or `Google-Extended` with `Disallow: /`. Compliance is voluntary but the major providers honour it.
Should I list my sitemap in robots.txt?
Yes. A `Sitemap:` line with the absolute URL helps every crawler find your URL list, and it works independently of Search Console submission.
What is the most common robots.txt mistake?
Deploying a staging file containing `Disallow: /` to production, which stops crawling of the entire site. Always read the generated file before uploading and re-check it after any deployment.