Prometheus
DE EN
DE EN

06 — Data model

How a record is built

Everything Prometheus stores and publishes is a row of the same shape: which project, which version, which cohort, which time window — and a value. For the reporting modes A to C that value is a band rather than a number; only mode D carries plain numbers, and only from public registries. The schema has no fields for people, hosts, or sessions — they are not left empty, they do not exist.

Key

The aggregate key

(project_id, version_id, cohort_id, window_start) — nothing more is needed to identify an aggregate. What the key leaves out is as much part of the statement as what it contains.

This is how a row comes about: from raw events that only ever live in memory, four key columns are set, the measured value is mapped onto a band, and only after the k-check does the result leave the machine.

  1. project_id pkg:npm/example-lib
  2. version_id 1.4.2
  3. cohort_id rt-runtime-mins
  4. window_start 2026-05-26T12:00:00Z
value 47 30-60 The exact minute count stays local; the band is what gets published.
k_effective 23 ≥ k_min 5 PASS
Example values. If k_effective were below 5, the same row would be rewritten locally into __small_cohorts — before sending, not after.

project_id — the PURL

Always a package URL, never free text: pkg:npm/example-lib, pkg:pypi/sample-pkg, pkg:maven/<group>/<artifact>, pkg:cargo/…, pkg:golang/…. That keeps pkg:npm/foo cleanly apart from pkg:pypi/foo.

version_id — the measured version

The version a measurement refers to. Only with it does the question become answerable whether a migration is actually landing in the field or whether an old major version keeps running.

cohort_id — the category

A predefined category, not a freely coined key — such as rt-runtime-mins, install-count, or dep-edges-from-public-manifests. New cohorts are published in the federation standard, not invented ad hoc.

window_start — the time window

Start of a tumbling window, UTC-aligned. Timestamps finer than the window granularity are forbidden by the schema — they would amount to a behavioural profile.

What is not part of the key: there is no tenant_id — the platform knows no tenants because it has no customers. And source_mode (A, B, C, D) is a column of its own, not a key component: the same cohort seen by the SDK and by the registry scraper sits side by side on purpose, instead of one overwriting the other. So the key groups, it does not enforce uniqueness — two rows with the same key and a different source_mode are two measurements of the same thing from two sources.

Wire format

How a submission is built

A submission is a canonically serialized JSON document with a window, a submitter identity, a signature, and a list of aggregates. What is not in here is not something a mirror gets from the submitter either — its own technical metadata and assessments it adds itself on ingress. 

{
  "schema_version": "1.0",
  "submitter_id":   "did:key:z6MkpTHR8VNsBxYAAWHut2Geadd9jSwuBV8xRoAnwWsdvktH",
  "signature":      "ed25519:5K3PqL…",
  "window_start":   "2026-05-26T12:00:00Z",
  "window_end":     "2026-05-26T13:00:00Z",
  "aggregates": [
    {
      "project_id":  "pkg:npm/example-lib",
      "version_id":  "1.4.2",
      "cohort_id":   "rt-runtime-mins",
      "metric":      "runtime_minutes_band",
      "value":       "30-60",      // a band, not a number
      "k_effective": 23,           // actual cohort size
      "k_min":       5,            // required floor
      "source_mode": "A"            // capture mode, a column not a key
    }
  ],
  "attestation": {
    "kind":   "declarative",
    "claims": ["k>=5", "no-pii-fields", "window-monotonic"]
  }
}

The example is a reading view of the structure, not the signed byte sequence: canonical JSON has no // comments, and JCS sorts the keys lexicographically. What gets signed is the result of that canonicalisation, not what is printed here.

Canonical bytes — RFC 8785

Sorted keys, no whitespace variants, UTF-8. The same input produces byte-identical output in every SDK language — without that, no signature would be verifiable across languages.

Signature — ed25519, no prehash

Signing happens directly over the canonical body bytes. The submitter identity is a did:key: a self-describing key, with no registration at any authority.

submission_hash — idempotency

sha256 over the same canonical body, but separate from the signature and never an input to it. It is the key in the public audit log: the same submission sent twice creates no second entry.

attestation — the assurance

The submitter states which conditions it upheld locally. In Phase 1 that assurance is declarative; from Phase 2 a cryptographic k-anonymity attestation takes its place.

Fields

The fields in detail

In modes A, B, and C the upper group comes from the submitter and is part of the signed bytes. Mode D has no submitter: there the mirror's own scraper fills the same columns mirror-internally and without a signature roundtrip. The lower group is set by the mirror in every case on ingress — it is assessment, not a claim by the submitter, and therefore deliberately unsigned.

Every field of an aggregate with example, meaning, and origin
Field Example Meaning Set by
From the submitter — from the mirror's scraper in mode D
project_id pkg:npm/example-lib Measured project as a PURL. Key component. Submitter (A–C) · scraper (D)
version_id 1.4.2 Measured version. Key component. Submitter (A–C) · scraper (D)
cohort_id rt-runtime-mins Predefined category. Key component. Submitter (A–C) · scraper (D)
window_start / window_end 2026-05-26T12:00:00Z Tumbling window, UTC-aligned. window_start is a key component. Submitter (A–C) · scraper (D)
metric runtime_minutes_band Which quantity was measured. Determines which bands apply to value. Submitter (A–C) · scraper (D)
value 30-60 Band label, not a plain number — the exception is mode D on public sources. Submitter · the scraper for D
k_effective 23 Actual cohort size behind the value. Empty for mode D, because a public registry has no cohort. Submitter · not applicable to D
k_min 5 Required floor. Default 5; at least 25 in the end-user-facing modes B and C. Submitter · not applicable to D
submitter_did
wire field: submitter_id		 did:key:z6Mk… Submitter identity as a self-describing key — an instance, not a person. Submitter · not applicable to D
source_mode A Capture mode A, B, C, or D. A column, not a key component. Submitter · the scraper for D
Set by the mirror on ingress
submission_hash 244be0…3df0 64 hex characters over the canonical bytes. Idempotency key and pointer into the audit log. Mirror
received_at 2026-05-26T13:04:11Z Arrival time at the mirror. Deliberately separate from the measurement window. Mirror
trust_tier new → established New submitters go through a seven-day burn-in before their data counts as established. Mirror
plausibility ok Anomaly marker (track-a-spike, track-d-missing). Nothing is deleted, it is annotated. Mirror
provenance registry-scraper@mirror-eu-1 Origin for mode D: which scraper of which mirror produced the row. Mirror
source_schema_version 1.0 Major version of the submission schema the row came from. Mirror

The four annotation columns — trust_tier, plausibility, provenance, and source_schema_version — are assessment output and simply do not exist in the wire format: a submitter can neither rate itself as trustworthy nor rewrite its own provenance.

Windows & bands

Banding instead of exact numbers

Two dials decide whether aggregates can be turned back into a profile: how finely time is resolved, and how precisely the value is stated. Both are set coarse on purpose.

Window sizes and where they become visible
Window Used for Visible in
1 min local aggregator only, for debugging never in a submission
1 h default for the self-instrumentation SDK (mode A) submission, API, snapshot
1 d default for registry scraping (mode D) mirror-internal record, API, snapshot
Bands per cohort and metric
Cohort / metric Bands
rt-runtime-mins
runtime_minutes_band		 0-5 · 5-15 · 15-30 · 30-60 · 60-180 · 180-1440 · 1440+
install-count
install_count_band		 1 · 2-5 · 5-25 · 25-100 · 100-1000 · 1000+
dep-edges-from-public-manifests
observed_dependency_count		 A plain number is permitted — mode D evaluates public manifests only, so there is nothing to protect here that is not already public.

Here is the same structure when it comes from registry scraping: no k_effective, because there is no cohort, but a provenance entry and a daily window instead. 

{
  "project_id":  "pkg:pypi/sample-pkg",
  "version_id":  "2.0.0",
  "cohort_id":   "dep-edges-from-public-manifests",
  "metric":      "observed_dependency_count",
  "value":       "1247",
  "k_effective": null,
  "k_min":       null,
  "window_start":"2026-05-26T00:00:00Z",
  "window_end":  "2026-05-27T00:00:00Z",
  "source_mode": "D",
  "provenance":  "registry-scraper@mirror-eu-1"
}

Absences

What the schema does not know

The following are not optional, not switchable, and not "internal only" — there is simply no field for them. An SDK bug that tried to send such a thing produces a rejected submission, not a record.

IP or MAC address Hostname, FQDN Username, email, OS user Process ID Host file paths Git paths, branch names Maintainer real names Environment variable contents Timestamps finer than the window

Two gates, not one

The local aggregator checks before anything leaves the machine, and the mirror checks again on ingress. Both checks are equally strict — the second exists for the case where the first was bypassed.

__small_cohorts

If a cohort stays below k_min, it is neither dropped nor quietly padded, but rewritten into a collective bucket. The value still counts towards the sum; the cohort disappears.

Rejection without a trace

If a submission violates the signature, the schema, the k floor, or window logic, the mirror answers with an error code — and persists nothing. Not even for debugging.

Window plausibility

window_start < window_end, and the window must not lie in the future; five minutes of clock drift are tolerated. That rules out post-dated submissions. The check has no lower bound so far — an old, correctly ordered window satisfies it.

Snapshots

Publication as Open Data

Whatever a mirror accepts gets published — no login, no paywall, and no second, richer edition for paying third parties. There is only this one body of data.

Properties of the published snapshots
Property Value
Schedule daily at 00:00 UTC
Formats Parquet as the primary format, plus CSV.gz — usable without Pandas or any other tooling
Variants raw holds the raw aggregates and is the canonical edition for auditing; resolved merges project aliases. Both are produced daily; __small_cohorts is never resolved by name, not even in resolved.
Integrity Both variants carry the same SHA256 manifest entry of the underlying data state plus the hash of the alias table used — including the raw variant, so it stays provable from the audit side what the resolution was made against.
License CC-BY-4.0 as a provisional decision; while legal clearance is pending, every snapshot carries a provisional clause in its manifest
Retention unlimited — older snapshots are archived, not deleted

Alongside the snapshots, the same structure is available live through the aggregate API and the event stream, likewise without a login. Anyone wanting to check whether two mirrors say the same thing compares their published audit log heads.

The data flow in 14 stages → Read the POC specification