# Method

## Domains and fetches

We took the Tranco top-1m list downloaded on 22 September 2026 and fetched
ranks 1 to 7,755, 7,749 domains after dropping duplicates. For each domain we
requested https://<domain>/llms.txt, /llms-full.txt and / on 23 September
2026 through a rotating residential proxy with a desktop Chrome user agent,
10 second timeout, redirects followed.

A file counts as a valid llms.txt when the response is 200, the body does not
start like HTML, the content type is text/plain, Markdown, octet-stream or
empty, and the body is over 20 bytes.

## Funnel

1. 7,749 domains requested.
2. 2,147 llms.txt requests failed (timeout, connection or TLS error). These
   domains are not retried in the main pass, so every count here is a floor.
3. 695 domains served a valid llms.txt; 104 served a valid llms-full.txt (3 of
   them without an llms.txt).
4. A pattern pass flagged 857 snippets on 133 domains: lines that address an
   AI system (AI, assistant, LLM, agent, model, ChatGPT, Claude, Gemini,
   Perplexity, Copilot) and use an imperative (recommend, should, must,
   always, never, do not, prefer, describe, prioritize, when asked), plus any
   heading such as "Instructions for AI" or "Guidance for AI systems".
5. All 133 domains were read in full and labelled: 16 steering, 29 route to own
   pages, 49 usage/licensing/agent, 39 product copy or false positive.
6. Homepages were checked for text addressed to AI inside hidden elements
   (display:none, zero font size, off-screen offsets, white text) and HTML
   comments: none found. No file contained "ignore previous instructions" or
   a variant.

## Checks

- Re-fetch: the llms.txt and llms-full.txt of all 45 steering and
  route-to-own-pages domains were fetched again on 23 September 2026. All 45
  still served a file. The steering text was present in all 16 steering
  files. One Wyndham response came back as a shorter file without the
  section; three further requests to www.wyndhamhotels.com/llms.txt returned
  the full file with the section.
- Variants: for the 184 domains in ranks 1 to 200 with no valid llms.txt at
  the bare domain, we requested www.<domain>/llms.txt and
  docs.<domain>/llms.txt. 7 valid files came back; one (www.kaspersky.com) has
  a steering section. It is included in quotes.csv and labels.csv but not in
  the 695-file counts.
- Templates: we grouped all 800 files by heading lines that mention AI and
  compared the 46 steering and route-to-own-pages files by shared 8-word
  shingles. The only large overlaps are sites with one owner (intercom.com,
  intercom.io and intercomcdn.com; marktplaats.nl and 2dehands.be; Hola's
  six domains). No steering file carries a generator comment or agency name.
  Web searches for the exact headings found no public template.
- Wayback Machine: for each steering company we listed CDX captures of
  <domain>/llms.txt and read each distinct 200 capture from the earliest
  until the section heading appeared. Seven of the 12 have no 200 capture.

## Limits

Bare domains only above rank 200, so files served only at www. or docs.
are missed there. Labels are one reader's judgement. The Wayback date is the
first capture that shows the text, not the day it was published. We did not
test whether any AI product reads these files or follows them.
