# Method

## Crawl date

The path fetches ran on 2026-09-04.

## Fetch method

Three paths were requested from every domain over plain HTTP through a
rotating proxy with a browser user agent: `/llms.txt`, `/llms-full.txt` and
`/robots.txt`. HTTPS was tried first and HTTP second, with a 15 second budget
per domain and the body capped at 200 KB. No page was rendered; these files
are served as text.

## Counting rule

A domain counts as serving an llms.txt only when the request returned 200
with a non-empty body that is not HTML. Servers that answer every path with
the homepage are dropped by content type and by a leading doctype. Blocking
counts only a full `Disallow: /` for a named AI crawler, so a site that
disallows one crawler from `/blog/` reads as not blocking.

## Generator rule

The generator is read from the first 3,000 characters of the body,
lowercased, and the first rule that matches wins:

1. `Shopify`: the body contains `agent instructions` and also either
   `shopify` or `ucp`.
2. `All in One SEO`: contains `all in one seo`.
3. `SiteSEO`: contains `siteseo`.
4. `Yoast SEO`: contains `yoast`.
5. `Rank Math`: contains `rank math` or `rankmath`.
6. `Webflow`: contains `webflow`.
7. `Framer`: contains `framer`.
8. `HubSpot`: contains `hubspot` in the first 400 characters.
9. `Mintlify`: contains `mintlify`.
10. `other generator`: matches the regular expression
    `generated (?:by|with)\s+([\w .-]{0,30})`.
11. `no signature`: none of the above.

A file that mentions one of these names for another reason is misread as
that generator, and a generator that does not print its name is counted as
no signature.

## Merge rule

One record per domain, holding all three paths from the same fetch. A domain
whose record already existed was not refetched.

## Floors

- A share is not published on a base under 20 domains.
- A count under 5 prints as "<5".

## Homepage limit

The files were read at the apex host only. A file at that host says nothing
about whether its contents are accurate, current or of any use to a model.

## Funnel

| Step | Inc 5000 | YC | Fortune 500 | Unicorns | All |
| --- | ---: | ---: | ---: | ---: | ---: |
| Domains on the list | 4,969 | 6,133 | 500 | 1,111 | 12,713 |
| Fetch attempted | 4,969 | 6,133 | 500 | 1,111 | 12,713 |
| robots.txt returned a status (base) | 4,931 | 5,630 | 459 | 1,058 | 12,078 |
| Dropped | 38 | 503 | 41 | 53 | 635 |

The 635 dropped domains, by the kind recorded against the robots.txt request:

| Drop reason | Inc 5000 | YC | Fortune 500 | Unicorns | All |
| --- | ---: | ---: | ---: | ---: | ---: |
| Connection failed through the proxy | 20 | 306 | 20 | 29 | 375 |
| Timeout or deadline | <5 | 85 | 19 | 20 | 128 |
| TLS failure | 12 | 104 | <5 | <5 | 122 |
| Redirect loop | <5 | 8 | 0 | 0 | 10 |

A connection failure covers a dead domain, a refused connection and a name
that did not resolve; the fetcher records one kind for all three, so those
cannot be separated. Domains that answered with a 403 are in the base, not in
this table, and count as not serving the file.

## Licence

Data: CC BY 4.0, Installmap, 2026-09-04.