Skip to content
Get the free beta

Identity & Join Keys

Documentation sections
On this page

How records identify and join

Atlas is a multi-dataset format: a single crawled page produces records spread across many datasets: its metadata, its links, its response, its accessibility findings, and more. Those records are tied together by a shared identifier, not by URL.

Join on pageId, never on URL

Every page-scoped record carries a pageId: a stable, time-ordered UUID assigned to one captured page. To assemble the full picture of a page, join its records across datasets on pageId.

page_metadata.pageId ─┬─ navigation.pageId
                       ├─ content.pageId
                       ├─ responses.pageId
                       ├─ accessibility.pageId
                       └─ … every page-scoped dataset

Do not match on URL. A URL can be redirected, normalized differently between datasets, or repeat across a site; pageId is the one stable identity for “this page, as captured in this archive.”

pageId is stable within an archive, not across crawls

The same URL crawled twice gets a different pageId each time. To line pages up across archives, for a before/after diff or a cross-mode comparison, match on the (normalized) URL, not pageId. Within a single archive, pageId is the join key; across archives, the URL is.

Other identifiers

  • crawlId: identifies the crawl run. Appears in the manifest and in operational datasets (events, sitemaps).
  • datasetId: a short identifier for the archive as a whole, used for compare/diff compatibility.

Datasets keyed differently

A few operational or record-scoped datasets use their own primary key instead of (or alongside) pageId:

Scroll horizontally to read all columns.

Dataset Primary key Link to a page
events eventId pageId (optional)
resources resId ownerPageId
sitemaps sitemapId none (carries crawlId)
assets assetId pageId
pipeline_failures / pipeline_alerts none keyed by URL + timestamp

The one to watch is resources: its page link is ownerPageId, not pageId. Asset IDs are deterministic, so the same asset on the same page is stable. See the dataset reference for the fields each record carries, and reading an archive for a worked join.