Architecture

RXDB Extractor separates database-specific meaning from generic extraction mechanics. The Python package owns inspection contracts, planning, normalization, identity reconstruction, relational validation, artifacts, and orchestration. A runtime bridge owns communication with RedEngine.

flowchart LR
    DB[RXDB / RBFX source family] --> BR[Runtime bridge]
    BR --> IN[Schema + capability inspection]
    IN --> PC[Profile compiler]
    PC --> PL[Query planner]
    PL --> BR
    BR --> NM[Mask-aware normalization]
    NM --> MG[Explicit-key batch merge]
    MG --> KY[Canonical key projection]
    KY --> VA[Count / PK / FK validation]
    VA --> AR[Parquet + manifests]
    AR --> OR[Checkpointed orchestration]

Runtime boundary

The maintained bridge is bridges/redatamx_bridge.R. Python communicates with it through protocol-v1 requests rather than importing or redistributing RedEngine.

Two transports implement the same logical boundary:

• one-shot JSON starts a subprocess per request;
• persistent JSON-lines keeps one bridge process alive per worker and can reuse opened database handles.

The bridge reports capabilities and schema, executes record plans, and returns structured rows/masks. Python remains responsible for deciding whether those results satisfy the extractor's invariants.

This boundary is intentional: a different bridge can implement the protocol without changing the relational extraction model.

Inspection and schema model

Inspection turns runtime metadata into the generic schema objects used by profile compilation. The important information is the entity graph, selectable entities, stored variables, aliases, and runtime capabilities.

A profile may supplement hierarchy information when upstream metadata is incomplete, but it may not silently contradict a parent relationship already established by inspection.

The result is a fail-closed adapter boundary: source-specific knowledge enters through explicit profile/schema evidence rather than through hidden conditionals in the generic engine.

Profile compiler

A portable JSON profile declares how one database maps onto the extraction model:

selection entity
identity scope
scope field
geography entities
record entities
generated ID fields
stored-variable policy
blocked ambiguous variables
relational keys
foreign keys

compile_profile() checks that contract against inspection and runtime capabilities, then produces a SliceSpec containing EntityJob and ForeignKeyJob objects.

Profiles are therefore adapters, not configuration sugar. A new database normally extends the system by adding and qualifying a profile rather than by forking the planner.

Planner and query execution

Each entity job is converted into one identity backbone plus bounded variable batches. The planner builds deterministic SPC using supported primitives:

SELECTION
NUMBER
direct ancestor inheritance
optional @cmpcode geography
FREQ

The executor boundary receives an ExtractionPlan and returns normalized record-shaped rows. The current runtime adapter translates the plan into protocol requests and the bridge executes the SPC.

Stored variables are deliberately batched. Structural identity dimensions repeat across batches so later combination never depends on row position.

Normalization and merge

Raw FREQ output contains masks and aggregate/margin cells. Normalization removes margins, requires concrete structural dimensions, enforces unit record counts, and rejects duplicate generated record IDs.

Variable batches are then merged by explicit record key. Compatible key sets are required before payload columns can be combined.

This is the main correctness boundary between hierarchical query output and relational records.

Identity projection

Generated source sequences are projected into canonical relational keys only after extraction has recovered the necessary hierarchy.

A canonical key combines a declared scope value with an exact positive integer sequence. The helper rejects values that could weaken identity semantics, including booleans, fractions, zero, negatives, and malformed numeric strings.

The key format is deterministic; its scientific meaning still depends on the qualified identity scope declared by the profile.

Dataset layer

run_slice() is the runtime-independent dataset assembly path. For each configured entity it:

1. executes the entity extraction;
2. projects declared canonical keys;
3. writes an atomic Parquet artifact;
4. emits configured count/primary-key checks;
5. evaluates declared foreign keys after all entities exist;
6. writes validation.json;
7. writes dataset-manifest.json with artifact/query/provenance evidence.

The executor callback is the only live-runtime boundary in this layer, which is why most correctness behavior can be exercised deterministically in Python tests.

Artifact and evidence layer

Parquet artifacts carry byte SHA-256 and schema fingerprints. The dataset manifest records selection, identity scope, scope source, entity artifacts, blocked variables, query hashes, expected counts, provenance, validation status, and a semantic hash.

Validation evidence is persisted separately so consumers can inspect the exact checks that ran.

See Outputs and Validation for the public artifact contract.

Orchestration and resume

run_partitions() repeats a compiled slice across an explicit partition inventory. Before reusing prior work it checks:

checkpoint identity
    +
entity Parquet hashes
    +
dataset-manifest semantic hash

The checkpoint identity binds source, schema, profile, runtime, extractor version, compiled slice contract, selection code, and expected counts.

A stale success checkpoint is invalidated before recomputation. A new success checkpoint is written only after validation passes.

The CLI's multi-worker mode partitions requests across independent bridge/runtime workers. Concurrency changes throughput/resource use, not extraction semantics.

Dependency direction

The intended dependency direction is:

source-specific project/profile
            │
            ▼
inspection + profile compiler
            │
            ▼
planner / runtime protocol
            │
            ▼
normalization + identity
            │
            ▼
dataset validation + artifacts
            │
            ▼
orchestration / CLI

Database-specific interpretation should not leak downward into generic extraction primitives. Conversely, source projects should not duplicate the generic identity/validation implementation.

What is deliberately outside the core

The architecture does not include direct RBFX decoding, binary patching, TABLE VIEW as a production primitive, bundled RedEngine distribution, natural-language querying, or a universal census ontology.

It also does not turn one qualified database into proof that every RXDB topology is supported. New sources enter through the extension/qualification process described in Extending RXDB Extractor.

For the algorithmic path, read Extraction and identity. For production behavior, continue to Operations and troubleshooting.