Skip to content

Documentation truth audit

Status: D0 complete for the MkDocs documentation wave
Audited baseline: main@c6beadd0d749cd3793b1f03c6c830b3bd03163f0 (2026-09-30)

Purpose

This audit separates current implementation truth from historical design targets before reader-facing documentation is expanded. It is an authoring control, not a substitute for user documentation.

Product boundary observed on the audited baseline

The implemented package is a Python CLI/library core for provenance-first relational extraction from supported hierarchical RXDB databases through a configurable RedEngine bridge.

The current CLI surface is:

  • rxdb inspect <database>;
  • rxdb extract <database> --profile <profile.json> --selection-code <code> --output <path>;
  • rxdb extract-many <database> --profile <profile.json> --partitions <inventory> --output-root <path>;
  • rxdb validate <output-or-validation.json>.

The runtime boundary is explicit. The Python process communicates with a bridge using either one-shot JSON subprocess requests or a persistent JSON-lines transport. The reference bridge is bridges/redatamx_bridge.R.

Implemented responsibilities and evidence owners

Area Current evidence owner Documentation consequence
CLI commands and options src/rxdb_extractor/cli.py, CLI tests Document the actual profile-driven CLI, not the older conceptual --entity target.
Runtime capabilities capabilities.py, runtime.py, bridge tests Describe capability guards as runtime behavior.
Bridge protocol/transports bridge.py, persistent_bridge.py, docs/BRIDGE_PROTOCOL.md, tests Keep protocol detail as internal/reference evidence; synthesize it for users.
Hierarchy and identity planning hierarchy.py, identity.py, planner.py, tests Explain generated IDs and inherited parent IDs without relying on row order.
Profile compilation profile.py, profile tests Treat profiles as the current extraction configuration boundary.
FREQ normalization normalizer.py, frequency.py, tests Explain complete non-margin cells and unit-count checks only to the extent implemented.
Explicit-key merge merge.py, merge tests Never describe positional batch assembly as supported behavior.
Dataset artifacts dataset.py, artifacts.py, manifest.py, tests Reference pages should derive names/semantics from these modules.
Validation validation.py, reports.py, validation/dataset tests Separate structural validation from broader source-equivalence evidence.
Checkpoint/resume checkpoint.py, orchestration.py, tests Completion is provenance/artifact aware; file existence is insufficient.
Partition inventory/workers partitions.py, orchestration.py, CLI tests Document bounded workers and explicit inventories from current CLI behavior.
Source identity source_identity.py, tests Explain source-family identity from implementation rather than generic hashing claims.

Strong claims supported by the repository

The following are suitable foundations for reader-facing explanation, subject to precise wording:

  1. Production extraction uses supported RedEngine query semantics centered on SELECTION, generated NUMBER identities, inherited ancestor IDs, and FREQ.
  2. Row position is not used as record identity or as the contract for merging variable batches.
  3. Variable batches merge by explicit generated record identifiers.
  4. Canonical persistence is Parquet in the implemented dataset path.
  5. Validation checks PK uniqueness, FK integrity and expected counts represented by the current extraction specification/reporting path.
  6. Resumable multi-partition orchestration verifies provenance and artifacts before skipping completed work.
  7. One-shot and persistent bridge transports implement the same bridge protocol boundary; persistent mode is opt-in.
  8. The Argentina Census 2022 corpus is the reference qualification case, while Argentina-specific source semantics are intended to remain outside the generic core.

Claims requiring qualification

Historical specifications contain targets, not only current interfaces

The specification bundle predates some implementation decisions. In particular, it includes conceptual targets such as:

  • rxdb extract ... --entity <ENTITY>;
  • a future rxdb.open(...) Python API;
  • broader validation-level switches;
  • possible native bindings.

Reader-facing documentation MUST NOT present these as current interfaces unless code has subsequently implemented them.

Identity scope

The historical ADR bundle marks processing partition versus canonical identity scope as an open design question. The current implementation has qualified record identity behavior for its supported profile/extraction path, but documentation must not silently generalize that into a universal cross-partition key guarantee without current evidence.

Runtime versions and performance

The README and evidence baseline record qualified Argentina laboratories and runtime/performance observations. These are evidence snapshots, not universal performance guarantees. Preserve date/runtime/corpus context when citing them.

Source-equivalence versus structural validation

The evidence baseline contains exact reaggregation experiments against selected Argentina variables. The generic rxdb validate command reads the persisted validation report; documentation must not imply that invoking it dynamically re-runs every historical source-equivalence experiment.

RedEngine distribution

Runtime use is implemented through an external bridge. Container/binary redistribution was historically left subject to licensing review. Documentation must not imply that RXDB Extractor redistributes RedEngine.

Public microdata distribution

The repository explicitly does not approve public redistribution of national person-level outputs by default. This remains outside the generic extraction mechanics and requires a separate disclosure/privacy decision.

Historical design corpus

The existing docs/spec/ bundle remains valuable as architecture history, normative design rationale, evidence baseline, and unresolved-question record. It should not be deleted or rewritten into the reader-facing site during D1.

Key sources:

  • docs/spec/00_START_HERE.md — bundle orientation and established feasibility evidence;
  • 01_PRODUCT_ARCHITECTURE.md — product layering and non-goals;
  • 02_GENERIC_EXTRACTOR_SPEC.md — original normative generic contract;
  • 04_VALIDATION_TEST_SPEC.md — validation philosophy and qualification fixtures;
  • 05_DATA_CONTRACTS.md — intended data/provenance contracts;
  • 07_ARCHITECTURE_DECISIONS.md — accepted and open ADRs;
  • 09_EVIDENCE_BASELINE.md — experimentally established baseline.

When those files disagree with current code about a user-visible interface, current code/tests own the description of implemented behavior; the older document remains evidence of intent/history.

Reader-facing information architecture

The D1 site deliberately separates reader tasks:

  • Getting started — a qualified first successful extraction;
  • Concepts — minimum RXDB/RedEngine/hierarchy mental model;
  • Architecture — implemented component boundaries;
  • Extraction and identity — central correctness mechanism;
  • Extending — profile/schema extension and acceptance expectations;
  • Operations — symptom-oriented recovery and resume;
  • Outputs — emitted data/provenance artifacts;
  • Validation — guarantees, evidence, and non-guarantees.

The existing specification corpus is not placed in the primary navigation. It remains in-repository for maintainers and provenance.

Authoring rules for later phases

  1. Current interfaces come from code and tests.
  2. Historical specs may explain intent but cannot override implemented behavior in user docs.
  3. Experimental evidence keeps its runtime, corpus, date, and scope.
  4. Open ADRs remain open unless new evidence resolves them.
  5. Argentina-specific source semantics belong in the Argentina repository.
  6. Generated/reference facts should point to or derive from authoritative code/schema where practical.
  7. Every runnable command added to a tutorial must be exercised before merge.
  8. No documentation page may turn a privacy/licensing caveat into an implicit approval.

D0 conclusion

The project is sufficiently mature for a reader-facing documentation layer. The principal documentation deficit is synthesis and navigation, not absence of technical contracts. D1 therefore adds a presentation/information-architecture shell without rewriting the existing specification corpus or prematurely filling later-phase pages.