Home / Method
Method
The sample, the fetch, the detection, and what the crawl leaves out.
Crawl: 2026-09-03
1. Sample
30,000 company domains were drawn from Crunchbase company records: companies with 11 to 1,000 employees, registered in the United States, the United Kingdom, Canada, Ireland, Australia, Germany, the Netherlands, France, Sweden, Norway, Denmark, Finland and Iceland. No industry holds more than 8% of the sample. Each company contributes one domain, its registered website with the www prefix removed.
2. Fetch
Each domain gets one request to its homepage over HTTPS, with HTTP as the fallback, through a rotating proxy with browser headers. Redirects are followed, the timeout is 12 seconds and the body is capped at 1.5 MB. A response that is not a 200 with an HTML body is recorded as a failure with its reason and is not counted.
3. Detection
The HTML, response headers and cookies of each fetched page are matched against 448 fingerprints: script hosts, generator meta tags, response headers, cookie names and asset paths on the site's own host. The fingerprints cover 26 software categories. One match records one install. A match shows that the site loads that tool's code. It does not show that the company pays for the tool.
4. Join and count
Every fetched domain is joined back to its Crunchbase company record, which carries the industry, country and employee band used for the cuts. Installs are rolled up per tool, then split by industry, employee band and country. A cell is published once its base reaches 20 crawled sites; below that it reads n/a. Findings on the stats pages compare only segments with at least 150 crawled sites.
5. Group crawls
The groups are crawled separately from the sample above. Each company on the list gets its homepage loaded in a headless browser, so scripts a tag manager injects after the page loads are counted. Where no rendered record exists, the plain fetch stands in. A group's denominator is every company on the list that returned a usable record. Where a list has not been crawled in full, the denominator counts against the domains fetched so far.
| Group | Companies on the list | Fetched | Rendered record | Plain record only | No record | Denominator | Crawl date |
|---|---|---|---|---|---|---|---|
| Inc 5000 (2026) | 4,969 | 4,929 | 4,814 | 72 | 43 | 4,886 | 4 September 2026 |
| Y Combinator companies | 6,133 | 6,133 | 5,242 | 144 | 747 | 5,386 | 4 September 2026 |
| Fortune 500 | 500 | 472 | 472 | 0 | 0 | 472 | 4 September 2026 |
| Unicorns | 1,111 | 1,000 | 1,000 | 0 | 0 | 1,000 | 4 September 2026 |
| SaaS companies | 55,839 | 9,008 | 56 | 8,117 | 835 | 8,173 | 4 September 2026 |
| Tranco top 10,000 | 10,000 | 1,573 | 1,572 | 0 | 1 | 1,572 | 4 September 2026 |
One row per crawl, including the wider sample, is on data updates.
Plain fetch against rendered fetch
Some groups were crawled both ways, so the gap between the two is measured rather than assumed. Across the domains with both records, a plain fetch finds this share of the installs the rendered page shows:
| Group | Domains with both records | Installs, plain | Installs, rendered | Plain recall |
|---|---|---|---|---|
| Inc 5000 (2026) | 3,569 | 13,100 | 36,546 | 35.8% |
| Y Combinator companies | 5,095 | 15,108 | 33,686 | 44.8% |
| SaaS companies | 13 | 72 | 154 | 46.8% |
Tools loaded through a tag manager are the largest part of the gap, which is why the group numbers and the wider sample numbers do not match for the same tool.
Not detectable from a homepage
Some software leaves no mark on a public homepage, so the crawl cannot count it and no share is published for it: server-side tracking that runs in the company's own backend, email marketing that only appears in sent mail, and anything behind a login. Payment processing is counted only where the vendor's script loads on the homepage itself, so a company that takes payment on a checkout page further into the site is not counted, and the payment shares are a floor rather than a total.
Attrition funnel
| Step | Count | Share of attempted | Note |
|---|---|---|---|
| Domains selected for the sample | 30,000 | 100.0% | Company domains drawn from the source universe. |
| Homepage fetched | 9,129 | 30.4% | A response was returned for the request. |
| Fetch succeeded | 8,291 | 27.6% | A 200 response with a readable HTML body. |
| Firmographics joined | 8,291 | 27.6% | Industry, country and employee band all present. The base every share on this site is divided by. |
Named failures
| Reason | Count | Share of attempted |
|---|---|---|
| HTTP 502 | 256 | 2.8% |
| HTTP 403 | 251 | 2.8% |
| Connection timeout | 209 | 2.3% |
| SSL error | 55 | 0.6% |
| HTTP 429 (rate limited) | 21 | 0.2% |
| HTTP 503 | 11 | 0.1% |
| HTTP 500 | 9 | 0.1% |
| Connection refused or reset | 5 | 0.1% |
| Proxy error | 4 | 0.0% |
| HTTP 401 | 4 | 0.0% |
| HTTP 526 | 3 | 0.0% |
| HTTP 530 | 2 | 0.0% |
| HTTP 521 | 2 | 0.0% |
| HTTP 525 | 2 | 0.0% |
| Too many redirects | 2 | 0.0% |
| HTTP 523 | 1 | 0.0% |
| HTTP 520 | 1 | 0.0% |
Detection method by signal type
- Script tag: the host a script element loads from, matched against the vendor's own domain
- Generator meta tag: the value of a generator meta tag, or an asset path only one CMS writes
- Response header: a header the vendor sets, or the CDN hostname a resource resolves to
- Form endpoint: the address a form on the page posts to
- Payment iframe host: the host a checkout iframe or payment script loads from
Out of scope
These tools are deliberately excluded from the crawl: they run behind a login or inside a data platform and leave no fingerprint on a public homepage, so a homepage crawl cannot count them.
- Salesforce: CRM used inside a login, no public homepage fingerprint
- Workday: HR and finance software, no public homepage fingerprint
- Snowflake: data warehouse, no public homepage fingerprint
- Gong: revenue intelligence, no public homepage fingerprint