Data flow — Prometheus v2

Translation. The German version of this document is canonical (CLAUDE.md, § Sprache & Tonalität). Where the two differ, the German text decides. Read the German original →

From the raw event in the end-user process (or in the registry crawl) to the Open Data snapshot. Each stage marks whether PII could theoretically pass through it and which defense-in-depth layer applies there.

1. Stage overview#

Data flow · 14 stages, two lanes (end user / mirror), two PII gates (k-anon, schema reject)

Stages in the end-user process (1–8):

RAW event (RAM, never on disk) — install · session-start · runtime-tick · feature-flag · dep-load.
Sanitize — drop or hash hosts, paths, env vars.
Cohort map — map identifiers onto (project_id, version_id, cohort_id) using local configuration.
Aggregate window — tumbling 1 h window (locally also 1 min for debugging).
k-anon guard — if k_effective < k_min, rewrite into cohort_id="__small_cohorts".
Schema-reject-PII — JSON schema validation; forbidden fields are rejected.
Sign + attestation — ed25519 signature; declarative claims (phase 1) / SNARK (phase 2).
Egress buffer — local disk, signed aggregates only; retry with exponential backoff; max 30 min before there is a risk of loss.

The transition happens over HTTPS POST /v1/submit (mTLS optional).

Stages at the mirror (9–14):

Submission validation — signature, schema, k_effective ≥ k_min, window plausibility.
Append-only log — Sigstore Rekor hash chain, publicly verifiable.
Aggregate store — ClickHouse, partition toYYYYMM(window_start).
Gossip / pull federation — mirror-to-mirror consistency replication.
Open Data snapshot — daily at 00:00 UTC, Parquet + CSV.gz, CC-BY-4.0.
Open Data API — read-only, no auth.

In parallel — mode D (registry scraping) runs mirror-internally and starts at stage 3 (cohort map onto known PURLs); stages 1 and 2 do not apply, because no private end-user events are read.

2. What happens where — responsibilities#

Stage	Responsible	Data protection layer
1 RAW EVENT	End-user process (RAM)	Never persisted. Discarded by the OS when the process ends.
2 SANITIZE	Local aggregator	First PII layer: known sensitive fields are emptied or hashed.
3 COHORT MAP	Local aggregator + maintainer config	Identifiers are mapped onto categories (`session-15-30min` instead of the value in the clear).
4 AGGREGATE WINDOW	Local aggregator	Reduced to counts/sums per window — no individual events any more.
5 K-ANON GUARD	Local aggregator	Second PII layer: cohorts below `k_min` are merged into `__small_cohorts`.
6 SCHEMA-REJECT-PII	Local aggregator	Third PII layer (defense in depth): the JSON schema rejects forbidden fields.
7 SIGN + ATTESTATION	Local aggregator	The signature evidences provenance; the attestation evidences the anonymity constraints.
8 EGRESS BUFFER	Local disk (signed aggregates only)	The data on disk contains no PII any more (stages 2–6 ran before it).
9 SUBMISSION VALIDATION	Mirror	Fourth PII layer (defense in depth): the mirror checks the schema and rejects PII.
10 APPEND-ONLY LOG	Mirror	Hash chain — contains no PII, because the aggregates were filtered beforehand.
11 AGGREGATE STORE	Mirror	ClickHouse partition. Public read.
12 GOSSIP	Mirror ↔ mirror	Replication of the same PII-free aggregates.
13 SNAPSHOT	Mirror	Daily Open Data artefact (Parquet/CSV.gz).
14 OPEN DATA API	Mirror	Read API without auth.

3. What never leaves the end-user process#

The following are permitted in no stage ≥ 7 — schema-reject-PII would turn them away, and that is defense in depth, not the primary filter:

IP address, MAC address, hostname, FQDN
Username, e-mail, OS user, process PID
File paths from the host filesystem
Git repository paths or branch names in the clear
Maintainer names or maintainer e-mails in the clear
The content of environment variables
Timestamps finer than the window granularity (typically 1 h)

What may be in there:

project_id (PURL — a public package identifier)
version_id (semver / similar — public)
cohort_id (a predefined category such as rt-runtime-mins, install-count)
Banded numeric values (30-60, 100-1000)
k_effective, k_min (audit fields)
Provenance: submitter_did, signature, mirror routing

4. Aggregate examples#

Mode A (self-instrumentation in the OSS lib pkg:npm/example-lib):

{
  "project_id": "pkg:npm/example-lib",
  "version_id": "1.4.2",
  "cohort_id":  "rt-runtime-mins",
  "metric":     "runtime_minutes_band",
  "value":      "30-60",
  "k_effective": 23,
  "k_min":       5,
  "window_start": "2026-05-26T12:00:00Z",
  "window_end":   "2026-05-26T13:00:00Z",
  "source_mode": "A"
}

Mode D (registry scraping over pkg:pypi/example-lib):

{
  "project_id": "pkg:pypi/example-lib",
  "version_id": "2.0.0",
  "cohort_id":  "dep-edges-from-public-manifests",
  "metric":     "observed_dependency_count",
  "value":      "1247",
  "k_effective": null,                     // Track A only reads public manifests, k-anon is trivial
  "k_min":       null,
  "window_start": "2026-05-26T00:00:00Z",
  "window_end":   "2026-05-27T00:00:00Z",
  "source_mode": "D"
}

5. Outage and replay behaviour#

Scenario	Behaviour
Mirror unreachable	The local aggregator buffers signed aggregates on local disk (stage 8). Retry with exponential backoff; after 30 min there is a risk of buffer overflow (`K5`).
Duplicate submission	The mirror already has the `submission_hash` in the audit log → idempotent 200 OK without a duplicate insert.
Cross-mirror divergence	The gossip job compares `audit_log.head`; on divergence a disclosure entry is set in the mirror's own `/v1/status`. Research can then export and compare both views.
Maintainer rotates their DID	The new DID is used in `submitter_id`; the old DID remains historically provable in the audit log. No migration needed.

6. Cross-references#

system-overview.md — component view
threat-model.md — STRIDE across this flow
03-risikoprofil.md — risks D-2-1 … S-2-3
05-zero-knowledge-vorschlag.md — ZK building blocks for hardening
02-poc-spezifikation.md — phase-1 MVP tracks A + B