Robots.txt Generator — Create robots.txt Online
Generate a valid robots.txt file to control how search engines crawl your website. 100% client-side — your data stays private.
What is a robots.txt file? A robots.txt file is a plain-text file served from the root of a site that tells crawlers which URL paths they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022, and consists of groups that each name a User-agent followed by Allow and Disallow path rules. It governs crawling, not indexing — a blocked URL can still be listed in search results.
How to Use the Robots.txt Generator
-
Start from a preset if one fits — Allow All produces the permissive default, Block All writes
Disallow: /, and the admin and image presets fill in the paths those cases usually need. A preset replaces the current rules rather than adding to them. -
Name the user-agent for each rule —
*covers every crawler that has no group of its own. Use a specific token such asGooglebotorBingbotonly when that crawler needs different treatment — and remember a named group replaces the wildcard group for that bot rather than adding to it. -
Enter paths, not URLs — Rules match the path portion beginning with a slash:
/admin/, nothttps://example.com/admin/. Matching is by prefix, so/adminalso blocks/administration— add the trailing slash when you mean the directory. -
Add an Allow rule to carve out an exception —
Allowexists to punch a hole in a broaderDisallow. Blocking/wp-content/while allowing/wp-content/uploads/is the usual shape. Google resolves conflicts by taking the longest matching rule. -
Add your sitemap URL — The
Sitemapdirective takes a full absolute URL and belongs to the file rather than to any one group, so it applies to every crawler no matter where it appears. -
Generate, then publish at the root — Copy or download the output and serve it at
https://yourdomain.com/robots.txtastext/plain. It has to be at the root: a file in a subdirectory is never read, and each subdomain needs its own.
How Crawlers Read robots.txt
Before a compliant crawler fetches anything else on a host it requests /robots.txt from that exact
location. The response code alone changes the outcome: a 200 means the rules are parsed and applied,
a 404 means no restrictions exist and the whole site may be crawled, and a persistent
5xx is treated by Google as a temporary instruction to crawl nothing at all — which is why a broken
server can quietly stall crawling until it recovers.
The file is scoped to one origin. Rules under https://example.com/robots.txt do not cover
https://shop.example.com/, and strictly speaking they do not cover the http:// version
either. Every host you want to control needs its own file at its own root.
Group Selection Is Winner-Takes-All
This is the part that trips people up. A crawler does not read every group that applies to it — it selects the
single most specific group whose User-agent token matches its name, and ignores all the others,
including *. So a file that disallows /private/ under User-agent: * and then
adds a User-agent: Googlebot group containing only a sitemap-friendly allow rule has just given Googlebot
free access to /private/. If a named crawler must obey a shared restriction, repeat that restriction
inside its own group.
Within the selected group, rules are matched against the URL path by prefix. Google resolves a conflict between an
Allow and a Disallow by taking the longest matching rule, and treats an exact tie in favour
of Allow. Paths are case-sensitive — /Admin/ and /admin/ are different rules —
while directive names are not.
Directive Reference
| Directive | Example | Meaning and support |
|---|---|---|
User-agent | User-agent: Googlebot | Opens a group. * is the fallback for crawlers with no group of their own. Token matching is case-insensitive |
Disallow | Disallow: /admin/ | Blocks any path starting with this prefix. An empty value blocks nothing; Disallow: / blocks the entire host |
Allow | Allow: /admin/help/ | Re-permits a subtree inside a blocked one. Only meaningful alongside a broader Disallow |
Sitemap | Sitemap: https://example.com/sitemap.xml | Absolute URL, independent of groups. May appear multiple times for multiple sitemaps |
* wildcard | Disallow: /*?sort= | Matches any sequence of characters — the standard way to block faceted or sorted URLs |
$ anchor | Disallow: /*.pdf$ | Anchors the match to the end of the URL, so this blocks PDFs without blocking /report.pdf.html |
Crawl-delay | Crawl-delay: 10 | Seconds between requests. Honoured by Bing and some others; ignored by Google, which sets its own rate |
# comment | # staging only | Everything after the hash on a line is ignored by parsers |
Keep the file small. RFC 9309 only requires a crawler to parse the first 500 kibibytes, and Google enforces exactly that limit, so rules beyond it may never be read. In practice a well-formed robots.txt is a few dozen lines.
robots.txt is not a security control and it is not a way to hide a page. The file is publicly
readable at a predictable URL, so listing /internal-admin/ there tells everyone the path exists — it is
one of the first places an attacker looks. Worse, blocking a URL prevents crawling but not indexing: if other sites
link to it, the URL can still appear in results, shown without a description because the crawler was never allowed
to read the page. To keep something out of search, allow crawling and serve a noindex meta tag or
header. To keep something private, require authentication.
Frequently Asked Questions
A robots.txt file is a plain text file placed at the root of your website that tells search engine crawlers which pages or directories they can or cannot access. It uses the Robots Exclusion Protocol.
No. robots.txt is a voluntary standard — well-behaved crawlers respect it, but malicious bots may ignore it. For true access control, use server-side authentication, firewalls, or IP blocking instead.
At the root of the host, served as text/plain with a 200 response — https://example.com/robots.txt and nowhere else. Crawlers request that exact path and never look in subdirectories. Each host is separate, so a subdomain such as shop.example.com needs its own file even if it shares the same site and codebase.
No, and this is the most expensive misunderstanding about the file. Disallow stops the crawler fetching the URL; it does not stop the URL being listed. If another site links to it, Google can index the address on the strength of that link and show it with no description, because it was never allowed to read the page. To remove a page from results, permit crawling and return a noindex meta tag or X-Robots-Tag header — a rule blocked in robots.txt means the noindex is never seen.
Two special characters are defined. * matches any run of characters, so Disallow: /*?sort= blocks every sorted variant of every listing page. $ anchors the match to the end of the URL, so Disallow: /*.pdf$ blocks PDF files but leaves /guide.pdf.html crawlable. Everything else is literal prefix matching, and paths are case-sensitive.
Check group selection first. A crawler obeys only the single most specific group that names it and ignores all others, so a Googlebot group anywhere in the file makes Googlebot skip the * group entirely — restrictions you meant to be global have to be repeated inside it. After that, check for a missing leading slash, a case mismatch in the path, and the file being served with an HTML content type or behind a redirect.
Generally no. Google renders pages the way a browser does, and blocking the stylesheets or scripts a page needs means it is assessed on a broken rendering, which can hurt more than the crawl budget it saves. Block genuinely low-value URL patterns instead — internal search results, infinite filter combinations, print variants, session-ID parameters.
No. The file is assembled in your browser and the download is generated locally; nothing you type is sent to a server. Only the sitemap URL is kept in the page address so a configuration can be reopened. Bear in mind that the finished file becomes public the moment you publish it, so treat its contents as visible to anyone.
Use Cases
Block Sensitive Areas
Prevent search engines from crawling admin panels, staging sites, or private directories that shouldn't appear in search results.
Optimize Crawl Budget
Direct crawlers away from low-value pages like search results, filters, and tag archives to focus on your important content.
E-Commerce Crawl Control
Block faceted navigation URLs and duplicate product pages to prevent crawler budget waste on an e-commerce site.
Staging Site Protection
Serve Disallow: / on a preview host so unfinished content never competes with production — while remembering that HTTP authentication, not this file, is what actually keeps visitors out.
Pointing Crawlers at Your Sitemap
Add a Sitemap line so any crawler that reads the file discovers your URL index without you submitting it to each search engine's console separately.