Validation
Validation is part of the extraction contract. A Parquet file existing on disk is not, by itself, evidence that the reconstructed relational slice satisfies its declared structural invariants.
What is checked
The exact check set is profile/request dependent. The current implementation can emit three kinds of dataset-level checks.
Expected entity counts
When an expected count is supplied for an entity, validation compares the number of reconstructed rows with that exact value.
A mismatch records observed and expected counts plus their delta.
Expected counts are optional. A profile/run without them does not silently invent a reference count.
Primary-key uniqueness
When an entity declares a primary key, validation checks that every row has a non-null value and that no value is duplicated.
The detail reports missing and duplicate counts.
This check applies to the projected relational key declared by the profile. Earlier extraction stages separately enforce generated record-ID and merge invariants.
Foreign-key integrity
For every declared relation, validation requires each child foreign-key value to be non-null and present in the parent key set.
The detail reports the number of invalid child values.
Pass/fail semantics
validation.json has status pass only when every emitted check passes. If there are no emitted checks, the current report model evaluates the empty set as passing; therefore a pass must always be interpreted together with the profile/manifest to understand which checks were actually configured.
For a single extract, the CLI returns exit code 0 for a passing report and 1 for a failing report.
For extract-many, a failed slice validation raises an orchestration error. No success checkpoint is written for that partition.
Earlier invariants fail before the report
Not every correctness rule appears as a named row in validation.json. The extraction pipeline also rejects invalid intermediate states before or while constructing the dataset, including:
- FREQ margin cells are removed using dimension masks;
- surviving structural dimensions must be concrete;
- a normalized record ID may not be duplicated;
- a surviving record cell must have
count == 1; - variable batches must have compatible explicit key sets before merging;
- canonical sequences must be exact positive integers;
- the profile must be compatible with the inspected hierarchy and runtime capabilities.
These failures normally stop extraction rather than becoming a validation.json check.
Reading a report
If the argument is a directory, the CLI reads validation.json inside it. You may also pass the report file directly.
The command prints the persisted JSON report. It returns 0 when status == "pass", 1 otherwise, and 2 for operational/input errors handled by the CLI.
Validate reads evidence; it does not rerun extraction
rxdb validate does not reopen RedEngine, regenerate queries, hash the source family, or re-execute the historical qualification experiments. It reports the validation result that was persisted during extraction.
What a passing report proves
A pass establishes that the checks present in that report succeeded for the reconstructed slice and that no earlier extraction invariant stopped the run.
With a typical relational profile containing primary keys and foreign keys, that includes uniqueness/non-nullness of declared PKs and referential integrity of declared FKs. With expected counts supplied, it also includes exact row-count agreement.
For resumable extract-many, reuse has an additional gate: the checkpoint identity must match and the persisted entity Parquet SHA-256 values plus dataset-manifest semantic hash must still verify.
What it does not prove
A passing report does not, by itself, prove:
- that every scientifically relevant variable was selected;
- that a stored source value is substantively correct;
- that labels/categories were interpreted correctly;
- that the chosen identity scope is valid for an unqualified database topology;
- equivalence to an independent official export unless that comparison was separately performed;
- national completeness when only selected partitions were requested;
- disclosure/privacy safety for person-level outputs;
- that an old report still corresponds to files that someone manually modified after extraction.
The last point is especially important for standalone rxdb validate: it reads the report and does not hash adjacent Parquet files. Artifact verification is performed by the extract-many resume path.
Expected counts as acceptance evidence
Partition inventories can carry exact expected counts. JSON entries use:
{
"selection_code": "061471101",
"expected_counts": {
"VIVIENDA": 73,
"HOGAR": 56,
"PERSONA": 137
}
}
CSV/TSV inventories can provide selection_code plus VIVIENDA/HOGAR/PERSONA values (including supported lower-case/count column variants).
Counts become part of the partition checkpoint identity. Changing an expected count therefore invalidates reuse and forces the partition through the current contract again.
Trust layers
It is useful to distinguish three layers:
The generic extractor owns the lower two layers and records runtime/source identities needed for reproducibility. Database-specific projects own the evidence that a particular source release, hierarchy, identity scope, and substantive interpretation are qualified for use.
For artifact structure, see Outputs. For failures and safe recovery, see Operations and troubleshooting.