122 of the 466 websites we could read (26.2%) served a real /llms.txt on 5 November 2026, out of the Tranco top 1,000 domains. 125 of the 488 robots.txt files we read (25.6%) fully block at least one AI crawler by name, GPTBot in 84 of them. This is a single run from one server in France, and the raw data per domain is a CSV.
Shotmint sells a screenshot and PDF API, so we have an interest in AI agents and crawlers, but no stake in whether llms.txt succeeds. Limits of the method are listed below.
| Tranco rank | Sites with llms.txt |
|---|---|
| 1-100 | 8 of 47 (17.0%) |
| 101-250 | 19 of 68 (27.9%) |
| 251-500 | 33 of 114 (28.9%) |
| 501-1000 | 62 of 237 (26.2%) |
| All | 122 of 466 (26.2%) |
The very top is not ahead: the top 100 has the lowest share. Reading the list below, it is mostly software, security, ad-tech and hosting companies, with few consumer sites. That is our impression from the names; we did not classify them. These are large sites, so this says nothing about the web as a whole.
# title, as the llmstxt.org proposal asks. The other 6 are text files that open with something else, such as a generator's comment. We counted those as present./llms-full.txt./llms.txt with HTTP 200 and an HTML page, usually their app shell or a soft 404. A status check alone would count them as having the file, nearly 4 extra sites for every 10 real ones.Publishing a file doesn't show that it gets used, and this census can't measure that. All we have is our own logs, one small site, so read this as an anecdote: from 17 October to 5 November 2026, shotmint.creasework.com/llms.txt and /llms-full.txt were fetched 35 times in total. ExaSearchBot made 12 of those. GPTBot and ChatGPT-User made none, and OAI-SearchBot, ClaudeBot and Googlebot one each. In the last 7 days alone, ChatGPT-User read our homepage 24 times. For our site, the live assistant traffic goes to ordinary pages.
All 122, in Tranco order, named by the host the domain landed on:
cloudflare.com, github.com, fastly.com, adobe.com, wordpress.org, digicert.com, opera.com, wordpress.com, sentry.io, intuit.com, samsung.com, dropbox.com, gravatar.com, ubuntu.com, shopify.com, paypal.com, kaspersky.ca, cisco.com, unity.com, forter.com, stripe.com, skyhighsecurity.com, hcaptcha.com, weather.com, roku.com, amplitude.com, sourceforge.net, hubspot.com, wix.com, datadoghq.com, salesforce.com, slack.com, liftoff.ai, centinelanalytica.com, appsflyer.com, magnite.com, zendesk.com, unisys.com, bilibili.com, ea.com, branch.io, dell.com, nvidia.com, checkpoint.com, paloaltonetworks.com, epsilon.com, cm.com, okta.com, about.gitlab.com, wps.com, name.com, plesk.com, dynatrace.com, atlassian.com, calendly.com, substack.com, dailymotion.com, businessinsider.com, ca.shein.com, trendmicro.com, braze.com, foxnews.com, taboola.com, selectel.ru, playstation.com, shop.com, klaviyo.com, capcut.com, dreamhost.com, cloudns.net, time.com, squarespace.com, newrelic.com, dailymail.com, digitalocean.com, vercel.com, target.com, sophos.com, unesco.org, contentsquare.com, intercom.com, onetrust.com, stackadapt.com, optimizely.com, redhat.com, surveymonkey.com, bitrix24.ru, fontawesome.com, gumgum.com, asus.com, supabase.com, mailchimp.com, zoho.com, yieldmo.com, moloco.com, webempresa.com, prnewswire.com, netlify.com, mixpanel.com, agora.io, auth0.com, life360.com, pendo.io, frontiersin.org, cloudinary.com, unpkg.com, cursor.com, porkbun.com, noip.com, skynet.ru, freewheel.com, lg.com, otto.de, wyze.com, classlink.com, instructure.com, tailscale.com, seedtag.com, repubblica.it, trendyol.com, flightradar24.com, clever.com.
Examples that returned a plain 404 for /llms.txt that day: microsoft.com, youtube.com, apple.com, amazon.com, wikipedia.org, whatsapp.com, bing.com, icloud.com, zoom.com, yahoo.com, roblox.com. Google, Facebook, Instagram, X and Reddit aren't in either list, because they refused the request or their robots.txt told us not to fetch.
Counted across 488 readable robots.txt files. "Blocked" means a group naming that crawler has Disallow: /. A partial rule doesn't count, and neither does a block that comes only from User-agent: *.
| Crawler | Fully blocked |
|---|---|
| Bytespider named by 102 | 89 (18.2%) |
| GPTBot named by 126 | 84 (17.2%) |
| CCBot named by 107 | 84 (17.2%) |
| ClaudeBot named by 119 | 80 (16.4%) |
| Google-Extended named by 106 | 67 (13.7%) |
| meta-externalagent named by 84 | 65 (13.3%) |
| Applebot-Extended named by 83 | 64 (13.1%) |
| PerplexityBot named by 113 | 63 (12.9%) |
| cohere-ai named by 68 | 58 (11.9%) |
| Amazonbot named by 73 | 57 (11.7%) |
| anthropic-ai named by 65 | 47 (9.6%) |
| ChatGPT-User named by 91 | 43 (8.8%) |
| Claude-User named by 67 | 42 (8.6%) |
| Claude-SearchBot named by 69 | 41 (8.4%) |
| Perplexity-User named by 61 | 40 (8.2%) |
| OAI-SearchBot named by 82 | 35 (7.2%) |
User-agent: *, on top of the table above.The list includes infrastructure domains (CDNs, ad servers) as well as ordinary sites, since Tranco ranks domains by traffic and not by whether they are websites. To check one site's llms.txt, use the llms.txt checker. To test one site against these rules, use the robots.txt checker.
Look at the content type as well as the status, because of the 46 soft 200s above:
curl -sL -o /dev/null -w '%{http_code} %{content_type}\n' https://example.com/llms.txtA real file answers 200 text/plain or text/markdown. 200 text/html means there is no file. Then request a made-up path such as /llms-a8f3.txt: if that also returns text of the same size, the server answers every path that way.
Across the same 466 homepages, from the raw HTML and response headers. Tags added by JavaScript after load aren't seen, so these are lower bounds.
| Found | Sites |
|---|---|
| <title> | 448 (96.1%) |
| viewport meta | 415 (89.1%) |
| <html lang> | 417 (89.5%) |
| favicon link | 395 (84.8%) |
| meta description | 380 (81.5%) |
| canonical link | 329 (70.6%) |
| og:title | 319 (68.5%) |
| og:description | 309 (66.3%) |
| og:image | 290 (62.2%) |
| twitter:card | 265 (56.9%) |
| JSON-LD | 221 (47.4%) |
| Strict-Transport-Security | 330 (70.8%) |
| Content-Security-Policy | 251 (53.9%) |
| X-Frame-Options | 251 (53.9%) |
A site without og:image shows up with no picture when shared. Our Open Graph checker shows what a page has, and the OG image generator makes the image.
/robots.txt, following redirects, matched on exact user-agent names only. A group with extra Allow paths next to Disallow: / still counts as blocked, unless it allows / itself.ShotmintCensus/1.0 with this page's URL, and made at most 6 requests per domain, not counting redirects.Data: top-1000-llms-robots-2026-11.csv has one row per domain. Free to reuse, with a link back to this page if you publish from it.
No signup, no card. Get a key in one click, then buy credits only if you need more (from €5 for 2,000, never expire). From a script, an AI agent, Make or Zapier, POST /v1/keys/free (no body) returns a key as JSON (openapi.json, no-code recipes).