RXDB and hierarchical data
RXDB Extractor exists because the data model exposed by an RXDB database is not an ordinary collection of flat tables. RedatamX databases describe entities arranged in a hierarchy. A census-like database might contain geography followed by dwellings, households, and people:
The names vary between databases. The important idea is structural: a person belongs to a household, a household belongs to a dwelling, and those records live below selectable geography.
The two shapes of the same data
RXDB Extractor translates between two useful representations.
The source is hierarchical:
The output is relational:
parent.parquet child.parquet
────────────── ─────────────
parent_key ◄──────── parent_key
... child_key
...
The difficult part is not writing Parquet. It is preserving enough source-native identity that the relational tables still describe the same hierarchy.
Entities, variables, and selectable ancestors
An entity is a node in the RXDB hierarchy. An entity can contain stored variables and can have parent and child entities.
A stored variable is a value already present on an entity in the source database. RXDB Extractor preserves source variable names in the extraction path rather than treating friendly labels as stable identifiers.
A selectable entity can be used by RedEngine to restrict work to one source partition. In the Argentina qualification corpus, geographic entities such as RADIO and FRAC are selectable while record entities such as VIVIENDA, HOGAR, and PERSONA are descendants of those selections.
This distinction matters. Selecting a small ancestor partition lets RedEngine traverse only the relevant descendants instead of scanning the entire database.
Selection is physical partitioning
The qualified extraction path uses RedEngine SELECTION as the physical partition primitive:
Selection answers where to extract. It does not by itself answer which record is which.
That second problem is record identity.
Records need explicit identity
A hierarchical engine can know that two values belong to the same underlying person without exposing a conventional database primary key. RXDB Extractor therefore asks RedEngine to generate deterministic entity sequences with NUMBER.
Conceptually:
DEFINE VIVIENDA.XVID AS NUMBER RADIO
DEFINE HOGAR.XHID AS NUMBER RADIO
DEFINE PERSONA.XPID AS NUMBER RADIO
For child records, the extractor also inherits identifiers directly from the ancestors that own them. A PERSONA record can therefore carry its own generated ID together with its HOGAR and VIVIENDA IDs.
This is the bridge from hierarchy to relational PK/FK structure.
Row order is not identity
RXDB Extractor does not assume that row 137 in one variable batch is the same source record as row 137 in another batch. Variable batches are joined by explicit generated identifiers.
Geography and composite keys
When the runtime supports it, the planner asks RedEngine for geographic compound codes using ENTITY@cmpcode. The implementation can combine a scope compound code with a positive generated sequence:
For example, 061471101:137 has the shape produced by the current canonical-key helper.
A key's meaning still depends on its declared identity scope. Do not infer a universal cross-partition identity guarantee merely from the string format. Historical design work explicitly treated processing partition versus identity scope as something to qualify with evidence.
Why FREQ can represent records
RedEngine's FREQ operation normally looks like an aggregation primitive. The extractor uses a qualified property of it: if the frequency dimensions include a unique generated record ID, then each complete, non-margin cell can correspond to one underlying record.
The planner therefore builds queries shaped like:
SELECTION <ancestor> == "<code>"
DEFINE <entity>.<own_id> AS NUMBER <scope>
DEFINE <entity>.<parent_id> AS <ancestor>.<generated_id>
DEFINE <entity>.<geo> AS <geography>@cmpcode TYPE STRING SIZE 32
FREQ <own_id> BY <parent_id> BY <geo> BY <stored variables...>
The exact generated query depends on the profile and runtime capabilities.
Margins, masks, and source missing states
A frequency result contains more than ordinary record cells. RedEngine masks identify margin/total cells. The normalizer discards cells whose relevant dimension mask marks a margin.
For a surviving record cell, RXDB Extractor requires:
- a concrete structural identity;
- no duplicate own ID;
- a unit frequency count;
- concrete hierarchy/geography dimensions where required.
Stored variables are different: a source variable can legitimately be missing or structural. Its non-margin mask can be preserved alongside the value as a companion __mask field.
This is why a frequency margin and a source missing value are not the same thing.
Profiles connect a database to the generic engine
The generic package does not hard-code one census hierarchy. A JSON extraction profile tells the compiler which source concepts to use: selection entity, identity scope, geography entities, generated ID fields, record entities, variable policy, blocked ambiguous variables, expected counts, and foreign-key relationships.
The profile is checked against runtime schema inspection. If the runtime already supplies hierarchy metadata, a profile cannot silently contradict it.
This boundary lets the extraction mechanics remain generic while database-specific semantics live outside the core.
The mental model
The shortest accurate model of RXDB Extractor is:
flowchart LR
A[RXDB hierarchy] --> B[Inspect schema and capabilities]
B --> C[Compile extraction profile]
C --> D[Select a source partition]
D --> E[Generate record and ancestor IDs]
E --> F[FREQ variable batches]
F --> G[Remove margins and enforce record invariants]
G --> H[Merge batches by explicit IDs]
H --> I[Project relational keys]
I --> J[Validate PK / FK / counts]
J --> K[Parquet + manifests + validation evidence]The product's core promise is therefore not "convert a file." It is recover relational records from a hierarchical query database while keeping identity, hierarchy, provenance, and validation explicit.
What this explanation does not claim
This page describes the implemented extraction model, not every historical target in docs/spec/. In particular it does not claim that a future native binding or conceptual Python API is currently available, that every RXDB topology is supported, that RedEngine is redistributed with the package, or that a successful structural validation reproduces every historical source-equivalence experiment.
Next, Extraction and identity follows the mechanism in more detail.