pakmath.com

SEO tools Robots.txt · Googlebot · Bingbot · AI crawlers

Robots.txt Generator & Tester

Build a robots.txt file from a CMS template or your own per-bot rules, check it for syntax errors, load any website’s live robots.txt, and test up to 50 URLs at once against Googlebot, Bingbot or AI crawlers. Matching follows Google’s documentation and the official Robots Exclusion Protocol (RFC 9309).

1. Generate robots.txt

Custom rules


    
    
    
  

2. Validate and test robots.txt

URLResultDeciding rule

How robots.txt rules are matched

  1. Pick the group. The crawler uses the group whose User-agent line names it most specifically (e.g. Googlebot-Image, then Googlebot), and only falls back to User-agent: * if none matches. Groups for the same crawler are combined.
  2. Find matching rules. Each Allow or Disallow path is compared with the start of the URL path (including the query string). * matches any characters and $ marks the end of the URL. Paths are case-sensitive.
  3. Longest match wins. The rule with the longest path decides. If an Allow and a Disallow are equally long, Allow wins. If nothing matches, the URL is allowed.

Robots.txt syntax reference

DirectiveExampleMeaning
User-agentUser-agent: GooglebotStarts a group of rules for one crawler; * means all crawlers.
DisallowDisallow: /cart/Do not crawl URLs starting with this path. An empty value allows everything.
AllowAllow: /cart/helpException inside a disallowed folder.
SitemapSitemap: https://example.com/sitemap.xmlFull URL of an XML sitemap; may appear anywhere, any number of times.
Crawl-delayCrawl-delay: 5Seconds between requests. Respected by Bing and Yandex, ignored by Google.
* and $Disallow: /*.pdf$Wildcard and end-of-URL anchor, supported by Google and Bing.

Common robots.txt mistakes

  • Blocking CSS and JavaScript. Google needs them to render pages; blocking them can hurt mobile-friendliness checks.
  • Using robots.txt to hide pages. Use a noindex meta tag for that; robots.txt only stops crawling.
  • Leaving Disallow: / after launch. Staging rules copied to the live site remove it from search. The SEO audit tool flags this automatically.
  • Blocking AI search bots by accident. A blanket AI block can hide you from ChatGPT search and Perplexity answers. Pair robots.txt with an llms.txt file if you want AI tools to understand your site.
  • Forgetting the Sitemap line. Listing your XML sitemap helps Google and Bing discover new pages faster.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain, so it opens at https://yourdomain.com/robots.txt. A robots.txt file in a sub-folder is ignored, and each subdomain (for example blog.yourdomain.com) needs its own file.

Does robots.txt stop a page appearing in Google?

Not reliably. Robots.txt controls crawling, not indexing. A blocked page can still be indexed from links pointing to it, without a description. To keep a page out of search results, allow crawling and add a noindex meta tag instead.

Which rule wins when Allow and Disallow both match?

Google and Bing use the most specific rule: the one with the longest matching path. If an Allow and a Disallow rule are equally long, Allow wins. This tester follows the same logic and shows which rule decided the result.

Does Google support Crawl-delay?

No. Googlebot ignores Crawl-delay; Google adjusts its crawl rate automatically. Bing does respect Crawl-delay, so it can still be useful for Bingbot on small servers.

Should I block AI crawlers?

Separate the two kinds. AI training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Meta-ExternalAgent) collect content for model training, and blocking them does not affect Google or Bing rankings. AI search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetch pages to cite them in AI answers; blocking them can remove you from ChatGPT search and Perplexity results.

How do I submit robots.txt to Google and Bing?

Upload it to your site root; search engines fetch it automatically, usually within a day. In Google Search Console you can view the fetched version in Settings › robots.txt and ask for a recrawl after changes; Bing Webmaster Tools has a robots.txt tester as well.

What is the maximum size of robots.txt?

Google reads the first 500 KiB of a robots.txt file and ignores anything after it. Keep the file short and use wildcards instead of listing thousands of URLs.

More free SEO tools