Extending RXDB Extractor

Supporting another RXDB database should normally be an adapter and qualification task, not a modification of the generic extraction algorithm.

The goal is to establish, with explicit evidence, that the new source hierarchy can satisfy the same identity, normalization, relational, and provenance contracts.

1. Start from runtime inspection

Before writing a profile, inspect the target database through the bridge:

rxdb --bridge "$RXDB_BRIDGE" inspect /absolute/path/to/database.rxdb

Capture the runtime version/capabilities and inspect the entity graph, selectable ancestors, stored variables, aliases, and metadata.

Do not infer a hierarchy from friendly variable names when runtime/schema evidence is available.

2. Choose processing and identity scopes

Identify a selectable ancestor suitable for bounded extraction and the scope within which generated record sequences are stable.

These are related but distinct concepts:

• selection entity controls which source partition is queried;
• identity scope defines the scope used for generated record identity;
• scope field carries the source value used when projecting canonical keys.

If @cmpcode is unavailable, the current compatibility path permits selection-code fallback only when selection entity and identity scope are the same. A source requiring something else needs new evidence/design, not a silent fallback.

3. Map the record hierarchy

For every relational record entity, identify:

• its source entity name;
• the generated own-ID field;
• record ancestors whose IDs must be inherited;
• source geography needed in the output;
• the intended primary key;
• parent relations that should become foreign keys.

The planner inherits each ancestor identifier directly from the entity that owns it. Do not relay an older ancestor ID through the nearest child merely because that is convenient in a profile.

4. Define variable policy

Profiles can select stored variables according to the implemented policy while explicitly blocking ambiguous variables.

Treat ambiguity as a qualification problem. If a source variable cannot be safely referenced because names/aliases collide, block it until the source-specific adapter has a justified resolution.

Do not rename source variables in the generic extraction path merely to make a profile prettier. Stable source identifiers are part of provenance.

5. Declare relational keys

Key projection converts exact generated sequences plus the declared scope field into deterministic relational keys.

A typical census-like relationship might be:

VIVIENDA(vivienda_key)
    ▲
    │
HOGAR(hogar_key, vivienda_key)
    ▲
    │
PERSONA(persona_key, hogar_key, vivienda_key)

Declare primary and foreign keys explicitly so the dataset layer can validate them.

6. Build a bounded qualification laboratory

Do not begin with the largest partition. Choose a small source selection whose structure can be independently inspected.

A useful qualification ladder is:

tiny known partition
        ↓
representative ordinary partition
        ↓
large / structurally demanding partition
        ↓
bounded multi-partition resume test

Where independently justified counts exist, put them in the partition inventory as expected_counts. They become validation checks and part of checkpoint identity.

7. Exercise the failure boundaries

A source adapter is more trustworthy when expected failures are tested as deliberately as successes.

At minimum, cover the relevant cases:

Boundary	Evidence to establish
hierarchy	invalid/contradictory parent maps are rejected
variables	unknown or ambiguous unblocked variables fail closed
identity	generated sequences are concrete, unique and valid
batching	payload batches merge only on compatible explicit key sets
geography	cmpcode or documented fallback matches the chosen scope
relations	declared PK/FK checks pass on qualification partitions
counts	independently known counts match where available
resume	unchanged work skips; changed provenance/artifacts recompute

Generic unit tests already cover the engine-level behavior. The new source project should add source-specific fixtures/evidence rather than duplicating every core test.

8. Keep source-specific semantics outside the core

A database adapter/project should own facts such as:

• source release/version and acquisition;
• legal/licensing/access constraints;
• concrete hierarchy and geography semantics;
• profile files and partition inventories;
• known anomalies/blocked variables;
• qualification laboratories and expected counts;
• substantive interpretation of variables;
• disclosure/privacy/release decisions.

The generic repository should own reusable mechanics and contracts.

The Argentina Census 2022 project is the first reference example of this separation.

9. When core code really must change

A core change is justified when the new database exposes a generic capability gap rather than a source-specific mapping problem—for example, a hierarchy shape the planner cannot represent while preserving explicit identity.

Before changing the core:

1. preserve a minimal failing fixture or protocol response;
2. state the missing generic invariant/capability;
3. add tests that fail for the right reason;
4. implement the smallest reusable extension;
5. rerun existing extraction/identity/validation tests;
6. requalify the source-specific adapter.

Do not weaken existing fail-closed checks to make one source pass.

10. Acceptance packet for a new source

Before describing a database as supported, retain enough evidence to answer:

Which exact source?
Which runtime?
Which profile?
Which hierarchy and identity scope?
Which variables were included/blocked?
Which partitions were exercised?
Which count/PK/FK checks passed?
Which limitations remain?
Can the same run be reproduced/resumed from preserved provenance?

The packet may live in the source-specific repository, but it should reference the generic extractor version/commit used.

Minimal profile shape

A profile commonly contains fields like:

{
  "name": "example",
  "source_database": "EXAMPLE",
  "selection_entity": "AREA",
  "identity_scope": "AREA",
  "scope_field": "XAREA",
  "geography_entities": ["AREA"],
  "id_fields": {
    "PARENT": "XPARENT",
    "CHILD": "XCHILD"
  },
  "entities": [
    {
      "entity": "PARENT",
      "own_id": "XPARENT",
      "variable_policy": "all-stored",
      "blocked_variables": [],
      "keys": [
        {"sequence_field": "XPARENT", "output_field": "parent_key"}
      ],
      "primary_key": "parent_key"
    }
  ],
  "foreign_keys": []
}

This is a structural example, not a drop-in profile. The target database's inspected schema and qualification evidence determine the real values.

For how the profile is compiled and executed, see Architecture. For the first live run, see Getting started. For production qualification and recovery, see Operations and troubleshooting.