# llms.txt, llms-full.txt and robots.txt across four company lists

Two kinds of row, marked by `row_type`. A `metric` row carries one aggregate
per group. A `domain` row carries one domain that served an llms.txt.

## Columns

- `row_type`: `metric` or `domain`.
- `group`: `inc5000_2026`, `yc`, `fortune500` or `unicorns`.
- `metric`: on a metric row, the name of the figure, such as `llms_txt_pct`,
  `base`, `blocks_any_ai_pct` or `disallow_pct_GPTBot`. Empty on domain rows.
- `value`: the figure. Counts are whole numbers, `_pct` metrics are
  percentages of that group's base. Empty on domain rows.
- `domain`: on a domain row, the domain that served an llms.txt. Empty on
  metric rows.
- `bytes`: the size of that domain's llms.txt body.
- `shopify_template`: `yes` when the file is the Shopify agent-instructions
  template, `no` otherwise.
- `blocks_any_ai`: `yes` when the domain's robots.txt has a full
  `Disallow: /` for at least one named AI crawler.
- `generator`: the software the file names, on domain rows only. One of
  `Shopify`, `All in One SEO`, `SiteSEO`, `Yoast SEO`, `Rank Math`,
  `Webflow`, `Framer`, `HubSpot`, `Mintlify`, `other generator` or
  `no signature`. METHOD.md holds the exact string rules. This is the
  column the report's generator table counts; `shopify_template` is the
  earlier flag and disagrees with `generator` on one Inc 5000 file.

On the Inc 5000, 877 of the 1,815 files carry a generator signature: 323
Yoast SEO, 315 Shopify, 134 All in One SEO, 91 Rank Math and 14 other.

## Denominators

The base for every percentage is the domains in that group whose robots.txt
request returned an HTTP status of any kind.

| Group | Base | Domains fetched |
| --- | ---: | ---: |
| Inc 5000 (2026) | 4,931 | 4,969 |
| Y Combinator | 5,630 | 6,133 |
| Fortune 500 | 459 | 500 |
| Unicorns | 1,058 | 1,111 |
| All four | 12,078 | 12,713 |

## Licence

Data: CC BY 4.0, Installmap, 2026-09-04.
Source: Installmap, September 2026.