robots.txt Generator & Tester
Assemble a robots.txt from user-agent groups, allow/disallow rules, crawl-delay and sitemap lines — then paste any robots.txt plus a URL and user-agent to see whether it is Allowed or Blocked, and which rule decided. Path matching follows Google's specification: wildcards, end-anchors, longest-match-wins, allow-beats-disallow. Runs entirely in your browser.
About this tool
Two panes over one shared model. The Build pane assembles a robots.txt from the parts it actually has: user-agent groups, an ordered list of Allow and Disallow rules per group, an optional Crawl-delay, and Sitemap lines that sit at the file level. The output updates as you type and is ready to copy to your site root. The Test pane is the inverse and the more useful half — paste any robots.txt (or load the one you just built), give it a URL and a user-agent, and it tells you whether that fetch is Allowed or Blocked and exactly which line decided it. The verdict uses Google's documented matching algorithm: the most specific user-agent group wins, then within that group the rule with the longest literal path match wins, and on a length tie an Allow beats a Disallow. The * wildcard and a single trailing $ anchor are honoured. Everything runs in your browser; no URL is fetched and nothing is uploaded.
When to use it
Questions
What is the difference between robots.txt, a robots meta tag, and the X-Robots-Tag header?
robots.txt controls crawling: whether a bot is allowed to request a URL at all. It lives at the domain root and is the first thing a crawler fetches. The robots meta tag (in a page's <head>) and the equivalent X-Robots-Tag HTTP header control indexing: whether a page that has been crawled may appear in search results, and how. The meta tag can carry directives robots.txt cannot express — noindex, nofollow, noarchive, max-snippet. Use robots.txt to keep bots out of areas that waste crawl budget or should never be fetched; use the meta tag or header to keep a crawlable page out of the index.
If I Disallow a page in robots.txt, does that remove it from Google?
No. Disallow stops Google fetching the page, but a URL it cannot fetch can still be indexed from external signals — links from other sites, the URL appearing in a sitemap. When that happens the result shows the bare URL with no title or snippet and a note that a description is not available. The trap is that because Google never fetched the page, it never saw any noindex directive on it. To actually keep a page out of results, allow crawling and add noindex (meta tag or X-Robots-Tag); once it has dropped out you can Disallow it if you also want to save the crawl.
Two rules match my URL — which one wins?
The one with the longer path pattern, counting literal characters and ignoring wildcards. Given `Disallow: /downloads/` and `Allow: /downloads/free/`, a request for /downloads/free/report.pdf is allowed, because /downloads/free/ (14 characters) is longer and more specific than /downloads/ (11). Rule order in the file does not matter. If the two matching patterns are the same literal length, an Allow beats a Disallow. This is Google's rule specifically; some older crawlers use first-match-wins instead, which is why rule order still gets treated as significant out of habit.
How do * and $ work in a path?
* matches any run of characters, including none, so `Disallow: /*.pdf` blocks every path that contains .pdf anywhere. $ anchors the pattern to the end of the URL, so `Disallow: /*.pdf$` blocks paths that end in .pdf but not /file.pdf?download=1. A path with no wildcard is a prefix match: `Disallow: /admin` blocks /admin, /admin/, /administrator and /admin-tools alike. If you mean only the directory, write `/admin/`. These two metacharacters are the only ones Google's parser recognises — everything else is literal.
How does a bot decide which User-agent group applies to it?
Each crawler picks the single most specific group whose user-agent token is a case-insensitive substring of its own name, and obeys only that group — directives are not inherited or merged across groups. Googlebot-Image uses a `Googlebot-Image` group if one exists, otherwise falls back to `Googlebot`, otherwise to `*`. So if you add a `Googlebot` group with a few rules, Googlebot stops reading the `*` group entirely, including any Disallow lines you left there. When you target a specific bot, copy the shared rules into its group.
Does Crawl-delay do anything?
It depends on the bot. Crawl-delay asks a crawler to wait a number of seconds between requests. Bing and Yandex honour it. Google ignores it completely — Googlebot sets its own crawl rate, adjustable only through crawl-rate settings in Search Console (and even that control has been wound down in favour of automatic rate-limiting). The tool emits the line because it is still valid syntax and other crawlers read it, but do not expect it to slow Googlebot.
What does the Sitemap line do, and does it belong to a group?
`Sitemap: https://example.com/sitemap.xml` tells any crawler where to find your sitemap. It is file-global — it applies regardless of which User-agent group it appears near, and by convention it goes at the top or bottom of the file, not inside a group. The URL must be absolute, including the protocol and host. You can list several Sitemap lines, one per sitemap or sitemap index. Listing it here is independent of submitting the sitemap in Search Console; doing both is fine and common.
Why test a rule before deploying it?
A robots.txt mistake is high-blast-radius and slow to notice. A stray `Disallow: /` on a line meant for a staging host, or a prefix rule that catches more than the directory you pictured, can pull large parts of a site out of crawling, and you often do not see the ranking drop until days later. Because the rules are terse and the matching is longest-match rather than top-to-bottom, the effect of a new line is not always obvious by reading. Pasting the file and checking a handful of real URLs — one you want blocked, a few you want kept — takes seconds and catches the over-broad rule before it ships.
Are robots.txt paths case-sensitive?
Yes. The path in a rule is matched against the URL path exactly, including case, because URL paths are themselves case-sensitive on most servers. `Disallow: /Private` does not block /private. Directive names (User-agent, Disallow, Allow) are case-insensitive, and so is the user-agent token when a bot matches its group — but the path is not. If your site serves the same content under mixed-case paths, you need a rule for each casing, or better, fix the duplicate URLs.