Documentation sections
On this page
How records identify and join
Atlas is a multi-dataset format: a single crawled page produces records spread across many datasets: its metadata, its links, its response, its accessibility findings, and more. Those records are tied together by a shared identifier, not by URL.
Join on pageId, never on URL
Every page-scoped record carries a pageId: a stable, time-ordered UUID assigned to one captured page. To assemble the full picture of a page, join its records across datasets on pageId.
page_metadata.pageId ─┬─ navigation.pageId
├─ content.pageId
├─ responses.pageId
├─ accessibility.pageId
└─ … every page-scoped dataset
Do not match on URL. A URL can be redirected, normalized differently between datasets, or repeat across a site; pageId is the one stable identity for “this page, as captured in this archive.”
pageId is stable within an archive, not across crawls
The same URL crawled twice gets a different pageId each time. To line pages up across archives, for a before/after diff or a cross-mode comparison, match on the (normalized) URL, not pageId. Within a single archive, pageId is the join key; across archives, the URL is.
Other identifiers
crawlId: identifies the crawl run. Appears in the manifest and in operational datasets (events,sitemaps).datasetId: a short identifier for the archive as a whole, used for compare/diff compatibility.
Datasets keyed differently
A few operational or record-scoped datasets use their own primary key instead of (or alongside) pageId:
Scroll horizontally to read all columns.
| Dataset | Primary key | Link to a page |
|---|---|---|
events |
eventId |
pageId (optional) |
resources |
resId |
ownerPageId |
sitemaps |
sitemapId |
none (carries crawlId) |
assets |
assetId |
pageId |
pipeline_failures / pipeline_alerts |
none | keyed by URL + timestamp |
The one to watch is resources: its page link is ownerPageId, not pageId. Asset IDs are deterministic, so the same asset on the same page is stable. See the dataset reference for the fields each record carries, and reading an archive for a worked join.