How robots.txt rules are matched
- Pick the group. The crawler uses the group whose
User-agentline names it most specifically (e.g.Googlebot-Image, thenGooglebot), and only falls back toUser-agent: *if none matches. Groups for the same crawler are combined. - Find matching rules. Each
AlloworDisallowpath is compared with the start of the URL path (including the query string).*matches any characters and$marks the end of the URL. Paths are case-sensitive. - Longest match wins. The rule with the longest path decides. If an Allow and a Disallow are equally long, Allow wins. If nothing matches, the URL is allowed.
Robots.txt syntax reference
| Directive | Example | Meaning |
|---|---|---|
| User-agent | User-agent: Googlebot | Starts a group of rules for one crawler; * means all crawlers. |
| Disallow | Disallow: /cart/ | Do not crawl URLs starting with this path. An empty value allows everything. |
| Allow | Allow: /cart/help | Exception inside a disallowed folder. |
| Sitemap | Sitemap: https://example.com/sitemap.xml | Full URL of an XML sitemap; may appear anywhere, any number of times. |
| Crawl-delay | Crawl-delay: 5 | Seconds between requests. Respected by Bing and Yandex, ignored by Google. |
* and $ | Disallow: /*.pdf$ | Wildcard and end-of-URL anchor, supported by Google and Bing. |
Common robots.txt mistakes
- Blocking CSS and JavaScript. Google needs them to render pages; blocking them can hurt mobile-friendliness checks.
- Using robots.txt to hide pages. Use a
noindexmeta tag for that; robots.txt only stops crawling. - Leaving
Disallow: /after launch. Staging rules copied to the live site remove it from search. The SEO audit tool flags this automatically. - Blocking AI search bots by accident. A blanket AI block can hide you from ChatGPT search and Perplexity answers. Pair robots.txt with an llms.txt file if you want AI tools to understand your site.
- Forgetting the Sitemap line. Listing your XML sitemap helps Google and Bing discover new pages faster.
Frequently asked questions
Where do I put robots.txt?
In the root of your domain, so it opens at https://yourdomain.com/robots.txt. A robots.txt file in a sub-folder is ignored, and each subdomain (for example blog.yourdomain.com) needs its own file.
Does robots.txt stop a page appearing in Google?
Not reliably. Robots.txt controls crawling, not indexing. A blocked page can still be indexed from links pointing to it, without a description. To keep a page out of search results, allow crawling and add a noindex meta tag instead.
Which rule wins when Allow and Disallow both match?
Google and Bing use the most specific rule: the one with the longest matching path. If an Allow and a Disallow rule are equally long, Allow wins. This tester follows the same logic and shows which rule decided the result.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay; Google adjusts its crawl rate automatically. Bing does respect Crawl-delay, so it can still be useful for Bingbot on small servers.
Should I block AI crawlers?
Separate the two kinds. AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Meta-ExternalAgent) collect content for model training, and blocking them does not affect Google or Bing rankings. AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch pages to cite them in AI answers; blocking them can remove you from ChatGPT search and Perplexity results.
How do I submit robots.txt to Google and Bing?
Upload it to your site root; search engines fetch it automatically, usually within a day. In Google Search Console you can view the fetched version in Settings › robots.txt and ask for a recrawl after changes; Bing Webmaster Tools has a robots.txt tester as well.
What is the maximum size of robots.txt?
Google reads the first 500 KiB of a robots.txt file and ignores anything after it. Keep the file short and use wildcards instead of listing thousands of URLs.
More free SEO tools
- SEO Audit ToolCrawl up to 25 pages and find broken links, title, description, heading, canonical, HTTPS and speed problems.
- SERP Snippet & Meta Tag CheckerPreview your Google result and check title and meta description length in characters and pixels.
- Word Counter & Keyword Density CheckerCount words, characters and sentences, find keyword and phrase density, and score readability.
- Meta Tag GeneratorCreate title, description, robots, canonical, Open Graph and Twitter tags ready to paste.
- Schema Markup GeneratorGenerate JSON-LD for Article, FAQ, Product, Local Business, Organization, Breadcrumb and more.
- Redirect & HTTP Status CheckerCheck status codes and full redirect chains for up to 10 URLs at once.
- Open Graph & Social Preview CheckerSee how a link looks on Facebook, X, LinkedIn and WhatsApp and find missing social tags.
- Heading Structure CheckerView any page’s H1–H6 outline and find skipped levels, missing or duplicate H1s.
- llms.txt GeneratorCreate an llms.txt file that guides ChatGPT, Claude and Perplexity to your best pages.
- All SEO toolsEvery free webmaster tool on pakmath.com in one place.