Extraction and identity
This page follows the implemented extraction path from a profile-defined slice to relational records. The central design constraint is simple:
every transformation that combines data must have an explicit identity contract.
1. Inspect before planning
The runtime bridge exposes database schema and engine capabilities to Python. The extractor checks the capabilities required for record extraction before compiling work.
Inspection provides the entity graph and variables. A profile can supplement parent relationships when an upstream metadata API is flat, but it cannot override a conflicting runtime hierarchy.
flowchart TD
DB[RXDB database] --> R[RedEngine bridge]
R --> S[Schema inspection]
R --> C[Capability inspection]
S --> P[Profile compiler]
C --> PThis keeps database semantics and runtime mechanics separate from the generic planner.
2. Compile a profile into entity jobs
The current CLI is profile-driven. compile_profile() checks the selected ancestor, identity scope, entity hierarchy, generated ID names, stored-variable policy, blocked ambiguous variables, expected counts, and declared foreign keys.
Each record entity becomes an extraction job. Stored variables are divided into bounded batches so a wide entity does not require one enormous frequency table.
Every batch repeats the same structural identity dimensions.
3. Generate IDs from the hierarchy
Suppose the inspected chain contains:
For a PERSONA batch, the hierarchy projection first defines generated IDs on relevant record ancestors. The target PERSONA then inherits each ancestor ID directly from the ancestor that owns it.
Conceptually:
DEFINE VIVIENDA.XVID AS NUMBER <scope>
DEFINE HOGAR.XHID AS NUMBER <scope>
DEFINE PERSONA.XPID AS NUMBER <scope>
DEFINE PERSONA.XVID AS VIVIENDA.XVID
DEFINE PERSONA.XHID AS HOGAR.XHID
Direct inheritance is deliberate. The implementation does not relay an older ancestor identifier through the nearest record parent.
4. Add source geography when supported
With the cmpcode capability, the planner adds source geographic codes:
If that capability is unavailable, profile compilation can use the selection code as the scope-field fallback for the supported path. The chosen geography-key mode is recorded in CLI provenance.
5. Emit one FREQ per variable batch
The planner constructs a deterministic SPC query whose dimensions include the own generated ID, inherited parent IDs, geography fields, and that batch's stored variables.
A simplified query is:
RUNDEF RXDB_ROWS
SELECTION RADIO == "061471101"
DEFINE VIVIENDA.XVID AS NUMBER RADIO
DEFINE HOGAR.XHID AS NUMBER RADIO
DEFINE PERSONA.XPID AS NUMBER RADIO
DEFINE PERSONA.XVID AS VIVIENDA.XVID
DEFINE PERSONA.XHID AS HOGAR.XHID
DEFINE PERSONA.XRADIO AS RADIO@cmpcode TYPE STRING SIZE 32
FREQ PERSONA.XPID BY PERSONA.XVID BY PERSONA.XHID BY PERSONA.XRADIO BY PERSONA.P02
This is illustrative SPC, not a command users need to author manually.
6. Normalize frequency cells into records
A raw FREQ result contains margin cells as well as complete cells. The normalizer uses the mask associated with every dimension and discards a row if any required dimension is a margin.
For every surviving row it enforces:
Violations are extraction failures rather than warnings.
Stored source variables may carry non-margin missing states. When requested, their masks survive as <field>__mask columns so a missing source value is not confused with a frequency margin.
7. Merge batches by explicit keys
An entity with many variables produces multiple normalized record batches:
The merge contract is the explicit record ID. The implementation requires compatible key sets and rejects duplicates or mismatches.
Positional concatenation is not a supported shortcut.
8. Project relational keys
The identity helper constructs deterministic composite keys from a declared scope compound code and an exact positive integer sequence:
Sequence normalization deliberately rejects booleans, fractions, zero, negatives, labels, and malformed numeric strings. That strictness prevents a transport representation from silently weakening identity semantics.
The dataset path then has enough information to expose own keys and parent keys for relational validation.
Scope is part of identity
The textual key format does not by itself prove that two extractions made under different processing partitions share the same canonical identity. Cross-partition stability depends on the qualified identity scope. Keep that scope explicit in profiles, manifests, and evidence.
9. Validate the reconstructed relations
Validation is not an afterthought. The dataset path checks the invariants represented by the extraction specification, including expected entity counts, primary-key uniqueness, and declared foreign-key integrity.
A successful run persists a validation report alongside data and manifest artifacts. The CLI validate command reads that persisted report; it does not re-execute every source query or every experiment recorded in the historical evidence baseline.
10. Persist provenance with the data
The extraction command records runtime capabilities, database metadata, profile identity, geography-key mode, and bridge transport in provenance. Multi-partition orchestration additionally binds source, schema, profile, and runtime hashes to checkpoint decisions.
This matters for resume. "The file already exists" is not enough to skip work: completion must still match the provenance/artifact contract.
11. The complete path
flowchart TD
A[RXDB + runtime] --> B[inspect]
B --> C[compile profile]
C --> D[entity jobs]
D --> E[variable batches]
E --> F[SELECTION]
F --> G[NUMBER + inherited IDs]
G --> H[FREQ]
H --> I[mask-aware normalization]
I --> J[explicit-key batch merge]
J --> K[composite PK/FK projection]
K --> L[structural validation]
L --> M[Parquet + manifests + validation]The same pattern is repeated for each requested record entity and, under extract-many, for each explicit source partition.
Boundaries to remember
RXDB Extractor currently uses an external RedEngine bridge; it does not decode RBFX storage directly. It does not make row order part of the data contract. It does not make TABLE VIEW a production dependency. It does not treat an output file as proof of completion. And it does not move database-specific interpretation—such as Argentina Census release semantics—into the generic extraction core.
For the upstream mental model, see RXDB and hierarchical data. For an executable first journey, continue to Getting started.