Outputs

RXDB Extractor writes a small evidence bundle for each extracted slice. The Parquet files are the data product; the JSON files make the extraction auditable and allow orchestration to decide whether prior work is still reusable.

Single-slice layout

For an extraction whose profile defines VIVIENDA, HOGAR, and PERSONA, the output directory has this shape:

<output>/
├── vivienda.parquet
├── hogar.parquet
├── persona.parquet
├── dataset-manifest.json
└── validation.json

Entity filenames are the lower-case entity names from the compiled profile. The exact set therefore depends on the profile; the three names above are the reference census-like case, not a hard-coded universal schema.

Writes of entity Parquet files, the dataset manifest, and validation report use temporary files followed by replacement so a partially written temporary artifact is not presented as the completed destination.

Entity Parquet files

Each entity artifact contains the normalized records reconstructed for that entity. Its columns can include:

the generated source sequence used during reconstruction;
inherited ancestor sequences;
source geography fields;
stored source variables selected by the profile;
optional <variable>__mask companions for preserved source missing/structural states;
projected canonical primary and foreign keys declared by the profile.

The extractor requires deterministic column order within an artifact and refuses to infer a Parquet schema from an empty row set.

The dataset manifest records four artifact facts for each entity:

Field	Meaning
`path`	path relative to the slice output directory
`rows`	Parquet row count
`columns`	ordered output column names
`sha256`	hash of the persisted Parquet bytes
`schema_hash`	semantic hash of field names, Arrow types, and nullability

The byte hash is used by resume verification. The schema hash is descriptive evidence of the written table schema.

`dataset-manifest.json`

The dataset manifest describes one extracted slice. Its current manifest_version is "1".

A representative shape is:

{
  "manifest_version": "1",
  "selection": {
    "entity": "RADIO",
    "code": "061471101"
  },
  "identity_scope": "RADIO",
  "scope_field": "XRADIO",
  "scope_source": "cmpcode",
  "batch_width": 5,
  "entities": {
    "PERSONA": {
      "artifact": {
        "path": "persona.parquet",
        "rows": 137,
        "columns": ["..."],
        "sha256": "...",
        "schema_hash": "..."
      },
      "blocked_variables": [],
      "query_hashes": ["..."],
      "expected_count": null
    }
  },
  "validation_status": "pass",
  "provenance": {},
  "semantic_hash": "..."
}

The manifest's semantic_hash is computed over the manifest contents before that field is added. It gives orchestration a compact identity for the persisted slice description.

query_hashes are semantic hashes of the generated SPC text for the entity's identity/payload plans. They allow the output to retain which query plans produced it without embedding every full query in the manifest.

blocked_variables records source variables that the profile intentionally excluded. expected_count is populated when the partition request supplied an expected count for that entity.

Provenance differs by command

For extract, provenance includes runtime capabilities, inspected database metadata, the profile hash, geography-key mode, and bridge transport.

For extract-many, each slice receives the run provenance used by checkpoint identity:

source_hash
schema_hash
profile_hash
runtime_hash
extractor_version
checkpoint_identity

These two commands therefore do not currently emit identical provenance objects. Consumers should read the fields actually present rather than assuming one universal provenance schema.

`validation.json`

The validation report is deliberately small:

{
  "status": "pass",
  "checks": [
    {
      "name": "unique:persona_key",
      "passed": true,
      "observed": 137,
      "expected": 137,
      "detail": "missing=0 duplicates=0"
    }
  ]
}

Its checks are generated from the compiled slice contract. See Validation for exact guarantees and limitations.

Multi-partition layout

extract-many creates one slice directory per selection code plus orchestration evidence at the root:

<output-root>/
├── run-manifest.json
├── .checkpoints/
│   ├── radio-061471101-<hash>.json
│   └── radio-064279901-<hash>.json
├── radio=061471101/
│   ├── vivienda.parquet
│   ├── hogar.parquet
│   ├── persona.parquet
│   ├── dataset-manifest.json
│   └── validation.json
└── radio=064279901/
    └── ...

Unsafe characters in a selection code are sanitized before they become directory/checkpoint components. The original selection code remains in the manifests/checkpoints.

`run-manifest.json`

The run manifest summarizes the requested orchestration invocation. It records the resolved database/output paths, partition inventory path and SHA-256, requested partition count, completed/skipped counts, actual worker count, source/profile/schema/runtime hashes, geography-key mode, bridge transport, and one outcome entry per partition.

A status of skipped means a prior checkpoint matched the current provenance/contract and its persisted Parquet hashes and dataset-manifest semantic hash still verified. It does not merely mean that a directory already existed.

The current run manifest is written after all requested partition outcomes complete successfully. If orchestration aborts on an error, do not assume a fresh root-level run manifest was written for that failed attempt; inspect the affected partition evidence instead.

Checkpoint files

Checkpoint JSON under .checkpoints/ is orchestration state, not a substitute for the dataset manifest. A complete slice checkpoint records:

checkpoint identity;
selection entity and code;
row counts by entity;
output SHA-256 by entity;
dataset-manifest semantic hash;
validation status.

A success checkpoint is written only after slice validation passes. Before recomputation, a stale success marker is invalidated so a failed attempt cannot leave an old success checkpoint behind.

What to preserve

For reproducible use, preserve the Parquet files together with their dataset-manifest.json and validation.json. For resumable multi-partition operation, also preserve the root .checkpoints/ directory and the partition inventory; run-manifest.json is useful run-level evidence.

Do not edit generated manifests or checkpoints to force resume. If an input or contract changed, allow the extractor to recompute the affected partition.

For the checks behind validation.json, continue to Validation. For resume and recovery behavior, see Operations and troubleshooting.