Paste a URL. We fetch the site's robots.txt and show whether Google, Bing and 12 AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) are allowed to crawl that page, which line decides it, and whether your sitemap works. Free, no signup.
The path of the URL you paste is the one tested (the home page if there's none).
| Which group applies | A crawler uses the group whose User-agent names it exactly. Only if none does, it falls back to User-agent: *. Groups are not combined, so a GPTBot group replaces your * rules for GPTBot. |
| Which rule wins | The longest matching Allow or Disallow path wins. On a tie, Allow wins. * matches anything and a trailing $ anchors the end of the URL. |
| Missing file | A 404 means "crawl everything". A 5xx server error makes Google treat the whole site as disallowed until it recovers. |
| Not a privacy tool | robots.txt is a public request, not access control. Blocked URLs can still be indexed from links. To keep a page out of search, use noindex. To keep it private, put it behind authentication. |
Training crawlers and search or browsing agents use different user agents, so you can opt out of model training and still show up when people ask ChatGPT, Claude or Perplexity:
# Opt out of AI training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /
# Everyone else, including search and AI answer engines
User-agent: *
Allow: /
Sitemap: https://your-site.com/sitemap.xmlGoogle-Extended and Applebot-Extended are opt-out tokens, not separate crawlers: blocking them doesn't affect Google Search or Apple's search crawler.
A crawler that's allowed in still has to render your page. The free website screenshot tool shows a page as a headless browser sees it, and the Open Graph checker and favicon checker cover the rest of your SEO head. To capture pages in code, one call to the Shotmint API:
curl "https://shotmint.creasework.com/v1/screenshot?url=https://example.com&full_page=true" \
-H "X-API-Key: YOUR_KEY" -o page.pngPay-as-you-go: €5 buys 2,000 renders and credits never expire. Get 100 free API renders