Getting started

This guide takes you from a fresh clone to the first validated real extraction. RXDB Extractor is the generic extraction engine: you provide an RXDB database, a working RedEngine bridge, and a profile that describes the database's supported hierarchy.

What the repository does not bundle

The repository does not contain RedEngine, a distributable RXDB microdata fixture, or one universal extraction profile. RedEngine has separate runtime/licensing constraints, and profiles are database-specific. The steps below make those boundaries explicit instead of pretending that a synthetic file proves real extraction.

1. Install the Python package

RXDB Extractor requires Python 3.10 or newer.

git clone https://github.com/matuteiglesias/rxdb-extractor.git
cd rxdb-extractor

python -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Confirm the CLI is installed:

rxdb --version
rxdb

The first command prints the package version. Running rxdb without a subcommand prints the command help.

Working on the repository?

Install python -m pip install -e '.[dev]' instead and run pytest. Documentation dependencies are deliberately separate in requirements-docs.txt.

2. Understand the runtime boundary

The Python package does not parse RedEngine's storage format directly. It sends protocol-v1 requests to a runtime bridge. The repository includes the reference R bridge:

bridges/redatamx_bridge.R

That bridge expects a compatible R/redatamx/RedEngine environment. Runtime installation and qualification are separate from installing the Python package.

Set the bridge command once your runtime is available:

export RXDB_BRIDGE="Rscript $(pwd)/bridges/redatamx_bridge.R"

You can use another protocol-compatible bridge, but the reference bridge is the maintained implementation in this repository.

3. Inspect a real database before extracting

Point the CLI at an RXDB database you are authorized to use:

export RXDB_DATABASE=/absolute/path/to/database.rxdb

rxdb --bridge "$RXDB_BRIDGE" inspect "$RXDB_DATABASE"

Inspection is the first meaningful runtime test. It asks the bridge for both capabilities and database schema.

A healthy response is JSON with two top-level sections:

{
  "capabilities": {
    "selection": true,
    "number": true,
    "inherited_define": true,
    "freq": true
  },
  "database": {
    "schema": {
      "entities": []
    },
    "metadata": {}
  }
}

The exact schema and additional capability fields depend on the database and runtime.

Record extraction requires selection, number, inherited_define, and freq. cmpcode is capability-dependent; the qualified compatibility path can operate without it only when the selection entity and identity scope are the same.

If inspection fails, do not start by changing the profile. First establish that the bridge can start, load its RedEngine environment, open the database, and return protocol-v1 JSON. See Operations and troubleshooting.

4. Obtain or define the database profile

Extraction is profile-driven. The profile is not a list of arbitrary columns: it is the adapter contract that connects one database's hierarchy to the generic extraction engine.

A profile declares concepts such as:

{
  "selection_entity": "<selectable ancestor>",
  "identity_scope": "<identity ancestor>",
  "scope_field": "<geography field>",
  "geography_entities": ["<ancestor>"],
  "id_fields": {
    "<record entity>": "<generated id field>"
  },
  "entities": [
    {
      "entity": "<record entity>",
      "own_id": "<generated id field>",
      "variable_policy": "all-stored"
    }
  ]
}

That fragment explains the shape; it is not a universal runnable profile. Real profiles may also declare hierarchy supplementation, blocked ambiguous variables, expected counts, key projection, and foreign-key relationships.

The profile compiler checks its claims against runtime inspection. A profile cannot silently contradict parent relationships that the runtime already knows.

Argentina Census 2022

If your source is the Argentina Census 2022 RXDB corpus, use the separately governed argentina-censo2022-rxdb adapter/project for source-specific profiles, provenance, controls, release semantics, and known anomalies. Do not reconstruct those assumptions from this generic tutorial.

5. Qualify one small selection

Before a large extraction, run one bounded source partition whose expected behavior you can inspect.

export RXDB_PROFILE=/absolute/path/to/profile.json
export RXDB_SELECTION=<selection-code>
export RXDB_OUTPUT=./out/qualification

rxdb --bridge "$RXDB_BRIDGE" \
  extract "$RXDB_DATABASE" \
  --profile "$RXDB_PROFILE" \
  --selection-code "$RXDB_SELECTION" \
  --output "$RXDB_OUTPUT"

On success the command prints a JSON summary pointing to the output directory, dataset manifest, and validation report. A validation failure returns a non-zero exit status.

The important part of this first run is not throughput. It is evidence that the database, runtime, profile, hierarchy, identity, variable batches, output artifacts, and validation contract agree.

6. Read the persisted validation result

rxdb validate "$RXDB_OUTPUT"

You can also point validate directly at a validation.json file.

This command reads the persisted validation report and exits successfully only when its status is pass. It does not reopen the RXDB and rerun every historical source-equivalence experiment.

After the first run, inspect the output directory rather than treating the presence of Parquet alone as completion. The dataset manifest and validation evidence are part of the result.

7. Move to resumable multi-partition extraction

Only after a bounded selection is qualified should you scale to an explicit partition inventory:

rxdb --bridge "$RXDB_BRIDGE" \
  --persistent-bridge \
  extract-many "$RXDB_DATABASE" \
  --profile "$RXDB_PROFILE" \
  --partitions /absolute/path/to/partitions.json \
  --output-root ./out/run \
  --limit 2 \
  --workers 1

Start with --limit and one worker. Each worker owns an independent bridge/RedEngine process, so increasing --workers increases runtime memory pressure as well as concurrency.

Once the bounded run is qualified, remove or raise --limit deliberately.

Re-running the same extract-many command uses provenance-aware checkpoints by default. A partition is skipped only when its checkpoint and artifacts still verify against the current source/schema/profile/runtime identities. Use --no-resume only when you intentionally want recomputation.

8. Choose the next documentation path

If you want to understand why the extraction works, read RXDB and hierarchical data and Extraction and identity.

If you are adapting another RXDB database, continue to Extending RXDB Extractor.

If you are operating long jobs, use Operations and troubleshooting, Outputs, and Validation.

First-run checklist

A first real extraction is qualified when all of the following are true:

• the Python package installs and rxdb --version works;
• the bridge starts in the intended runtime;
• inspect reports the database schema and launch-critical capabilities;
• the profile matches the inspected hierarchy;
• one bounded selection completes;
• the extraction command reports validation status pass;
• rxdb validate <output> reads the persisted passing report;
• the output retains its manifest/provenance evidence alongside Parquet.

That is the minimum useful journey. National-scale extraction, new database support, runtime packaging, and public redistribution are separate decisions.