Test robots.txt rules against any user agent, including AI bots. Runs in your browser — nothing is stored or sent anywhere.
Paste your robots.txt content, enter a URL path to test, and select a user-agent. The tool parses all User-agent groups, Allow and Disallow directives, then matches your test path following RFC 9309 — the most specific rule (longest matching path prefix) wins. Everything runs in your browser.
When multiple rules match a path, the rule with the longest matching pattern takes precedence — not the order in the file. An Allow and a Disallow of equal specificity resolves to Allow. This is the standard that Google, Bing and most crawlers follow.
AI crawlers like GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and CCBot (Common Crawl) respect robots.txt. Many publishers block some or all of these bots. Use the preset buttons to quickly check whether your robots.txt allows or blocks each one — no need to remember exact user-agent strings.
The example above blocks GPTBot from your entire site. To allow GPTBot on specific paths while blocking the rest, add Allow directives with more-specific paths — they will win over the broad Disallow per RFC 9309.
Clever Marketing