Crawl Presence Registry Dataset

Before any AI trains on the open web, a crawler has to have kept a copy. This dataset is the standing record of what the open crawl actually holds about our company — collection by collection, capture by capture, era by era — and the robots posture, recorded as our owner’s decision, under which that crawling happens. “The corpus knows us” is a query result here, not an impression.

The standing registry of the training/research-corpora element for The Flat Rate Movers: every Common Crawl collection’s recorded answer for this domain — captures classified by URL class and content era, with the query state and method on every collection — the recorded robots posture under which crawling happens (owner decision, 2026-10-31: Option 1, open training), the signed signal-observation log that proves what the estate’s robots file said and when, and the element’s gap records P1–P6 each with its designed solution. Query it three ways: the full JSON file; ?table= on the API route; and the ?table= form of /v1/datasets for the estate inventory.

One JSON file: 129 collections · 16197 capture rows · 14822 with status 200 · current-estate captures: 0 · 17 robots agent groups (7 training, 9 retrieval) · 6 gaps (2 resolved). Generated from a live sweep of the Common Crawl index by the Flat Rate Machine — never hand-kept.

Download datasets-crawl-presence.json API: /v1/crawl-presence

What this dataset contains

The headline finding: a zero that is a fact

Every collection Common Crawl has published to date was crawled before our current site existed — the estate cut over on October 28, 2026, and the newest collection in the index predates it. So the current estate is present in zero collections, and everything the crawl holds about this domain — 16197 captures across 129 collections — describes predecessor sites: the WordPress site that ran until the cutover, the Wix-era site before it, and earlier forms. Each capture row carries its content era, computed from its timestamp against the cutover and its URL form, never hand-tagged; 0 rows are flagged for era review because their form contradicts their timestamp. The first collection crawled after the cutover is the current estate’s corpus debut, and this registry is the instrument standing watch for it (gap P1, resolving in batch E5-B2).

Every collection, with its recorded answer

The seed sweep of October 31, 2026 queried every collection in Common Crawl’s index for this domain, 2026 series first, under a retry discipline. A collection that returned captures is queried-clean; one whose index could not be read after all retries stands as unverifiable-index-error — a visible state, never a silent zero (gap P6). Capture windows are read from this domain’s own rows; where a domain returned no captures, no window is derivable and none is invented.

The robots posture — a recorded decision

Our robots.txt posture is not an accident of a default file. On October 31, 2026 the owner decided it: Option 1 — open training, made deliberate (Dossier 05 §4, gap P3). Option 1 changes no file: the live robots.txt stands exactly as served. What changed is that the decision is now recorded here with the file’s digest, and two gates keep the record and the file in lockstep — this dataset’s generator refuses to build if the served file’s digest ever differs from the deployed source, and the estate’s verification suite re-checks the recorded decision against the live file on every run.

Decision

Option 1 — open training (decided by the owner, 2026-10-31); file change: none

Served file

https://theflatratemovers.com/robots.txt · 958 bytes · SHA-256 94dea4b3f3e7806c34c1984f54948326f83027607ff95d155012fe7912ceac80 (observed 2026-10-31T21:04:17Z, HTTP 200)

Drift gate

no-drift-observed — served bytes equal the deployed source file’s bytes; no Cloudflare-prepended directives observed in the served file

Content Signals

search: unset · ai-input: unset · ai-train: unset (absent from the served file)

__usage_ai-training token

absent — recorded as decision evidence in the dataset; not deployed, because the owner chose open training

No archive-specific agent is named in the file; archive admission rides on the wildcard group and on CCBot, whose collections double as the public archive.

The signed signal log

Preference signals only matter as served, so the registry keeps a signed observation log: each entry records a signal exactly as it was served — digests, byte size, parsed directives, validity window — chained to the previous entry and signed with the estate’s Ed25519 key, verifiable against the public key file. The first entry observes today’s robots.txt. No Internet Archive capture of the current robots.txt exists yet (the captures on file are of the predecessor file, March 2026), so its archive field stands honestly absent; the optional RFC 3161 timestamp field stands unpopulated — no Time-Stamp Authority has been adopted.

Entry

sig-2026-10-31-robots-001 · mechanism robots · observed 2026-10-31T21:04:17Z by Flat Rate Machine signal observation (Mira) — urllib fetch + build-crawl-presence parser, 2026-10-31

Body

SHA-256 94dea4b3f3e7806c34c1984f54948326f83027607ff95d155012fe7912ceac80 · 958 bytes · 18 parsed directives · spec RFC 9309

Entry digest

e6927f3889fe39008bad8fac1ccad36029ea1cf726861cb3c237f60510bbdde7 · Ed25519, key theflatratemovers-2026-10-29-1 · previous entry: genesis (none)

Gaps P1–P6, with their solutions

Six named gaps. Each record in the dataset carries its evidence and impact in full; the designed solution is printed here in full — solutions, not problem lists.

Mechanism pins — the facts the decision rests on

Pinned against primary sources in the 2026-10-31 crosscheck reconciliation, so the posture is argued from verified ground.

Facts & download

File

datasets-crawl-presence.json · 7,708,560 bytes · SHA-256 042dfa089e24c53b4636ccfd722f65ea1a0173421b59aa9d03a8f2ca72fef6b0

Contents

129 collections · 16197 capture rows · 14822 with status 200 · current-estate captures: 0 · 17 robots agent groups (7 training, 9 retrieval) · 6 gaps (2 resolved)

License

CC BY 4.0 — reuse with attribution to The Flat Rate Movers

API

GET https://api.theflatratemovers.com/v1/crawl-presence (byte-identical file; ?table= filters to collections, presence, summary, robots, signals, inspections, releases, gaps or mechanisms)

Provenance

Sealed in the signed AI master index (Ed25519). Verify any copy: GET https://api.theflatratemovers.com/v1/verify?file=datasets-crawl-presence.json

Download the dataset (JSON)