Skip to content

Operations and troubleshooting

This guide is for running real extractions safely: qualifying a runtime, scaling across partitions, understanding resume decisions, and recovering from failures without weakening provenance.

The operational rule is:

diagnose from the observable failure, preserve evidence, and let provenance-aware recomputation repair stale work.

Do not mark work complete by creating files or editing checkpoints manually.

Before a long run

Start with a bounded qualification:

rxdb --bridge "$RXDB_BRIDGE" inspect "$RXDB_DATABASE"

rxdb --bridge "$RXDB_BRIDGE" \
  extract "$RXDB_DATABASE" \
  --profile "$RXDB_PROFILE" \
  --selection-code "$RXDB_SELECTION" \
  --output ./out/qualification

Confirm that inspection reports the launch-critical capabilities and that the bounded slice passes validation.

For a partitioned run, start conservatively:

rxdb --bridge "$RXDB_BRIDGE" \
  --persistent-bridge \
  extract-many "$RXDB_DATABASE" \
  --profile "$RXDB_PROFILE" \
  --partitions "$RXDB_PARTITIONS" \
  --output-root ./out/run \
  --limit 2 \
  --workers 1

Only increase the limit or worker count after the target runtime behaves correctly on representative partitions.

One-shot vs persistent bridge

The reference bridge supports two protocol-v1 transports.

One-shot JSON starts a bridge subprocess for each request. It is the simplest compatibility/debugging path.

Persistent JSON-lines keeps one bridge process alive and can keep opened RXDB handles cached across requests. Enable it with --persistent-bridge.

The query semantics are intended to be identical. Persistent mode is an operational optimization, not a different extraction contract.

When troubleshooting a persistent-only failure, rerunning a small partition in one-shot mode is a useful isolation step: it distinguishes extraction/query problems from process-lifetime/transport problems without changing the profile.

Worker sizing

extract-many --workers N uses independent workers. Each worker owns its own bridge/RedEngine process; requests are not multiplexed concurrently through one persistent process.

The CLI accepts 1–32 workers, but that is an argument bound, not a resource recommendation. Actual safe concurrency depends on the target runtime, database, partition sizes, and host memory.

Start at one worker. Increase deliberately while observing memory and runtime stability. The historical qualification numbers in the project README describe one preserved workload; they are not a universal sizing formula.

If there are fewer partitions than requested workers, the actual worker count is reduced to the partition count and recorded in run-manifest.json.

Partition inventories

extract-many requires an explicit inventory. Supported forms are:

  • JSON: strings or objects with selection_code and optional expected_counts;
  • CSV/TSV: a required selection_code column and optional supported entity-count columns;
  • text: one selection code per line, with blank lines and # comments ignored.

Duplicate selection codes are rejected.

Use expected counts when you have an independently justified acceptance count. Do not add guessed counts merely to populate validation.

How resume decides to skip

Resume is enabled by default. A partition is skipped only if both contract identity and persisted artifacts still verify.

The checkpoint identity includes:

run provenance
  ├── source hash
  ├── schema hash
  ├── profile hash
  ├── runtime hash
  └── extractor version

compiled slice contract
selection code
expected counts

The compiled contract includes the selection/identity configuration, batch width, cmpcode mode, entity variables, generated/inherited IDs, geography entities, blocked variables, projected keys, primary keys, and foreign keys.

After the identity matches, the runner verifies every checkpointed entity Parquet SHA-256 and the dataset-manifest semantic hash.

flowchart TD
    A[Partition requested] --> B{resume enabled?}
    B -- no --> R[recompute]
    B -- yes --> C{checkpoint identity matches?}
    C -- no --> R
    C -- yes --> D{Parquet hashes verify?}
    D -- no --> R
    D -- yes --> E{manifest hash verifies?}
    E -- no --> R
    E -- yes --> S[skip partition]

File existence alone never triggers a skip.

What invalidates reuse

Expect recomputation when a checkpoint is absent/corrupt/incomplete, when its identity changes, or when verified output artifacts no longer match.

Identity changes include changes to the source family hash, inspected schema, profile, runtime capabilities, extractor version, compiled contract, selection code, or expected counts.

Artifact corruption also invalidates reuse even if the checkpoint identity itself still matches.

--no-resume bypasses valid checkpoint reuse and recomputes every requested partition. Use it for intentional recomputation, not as the first response to an unexplained failure.

Source identity

Unless --source-hash is supplied, extract-many computes source identity from the RXDB file and adjacent RBFX shards whose filenames share the RXDB stem. The identity incorporates each member's filename, size, and SHA-256.

This can be expensive for a large source family, but it prevents a changed shard from being treated as the same source.

--source-hash is an expert escape hatch for a precomputed exact source-family identity. Supplying an arbitrary label weakens the resume contract; use it only when an external process has computed the intended exact identity.

Failure: runtime not configured

Symptom:

{"command": "inspect", "status": "runtime-not-configured"}

The CLI had no --bridge command. Configure the bridge explicitly.

rxdb --bridge "$RXDB_BRIDGE" inspect "$RXDB_DATABASE"

Exit code is 2.

Failure: bridge does not start or times out

Check the bridge command outside a large run first. Verify the executable/R environment exists and the target RedEngine dependencies are available.

The default per-request bridge timeout is 120 seconds and can be changed with --bridge-timeout. Raising it is appropriate only when the request is legitimately slow; it does not fix a hung or incompatible runtime.

If persistent mode is involved, reduce to a small one-shot inspection/extraction to isolate transport lifetime from extraction semantics.

Failure: missing record capabilities

Record extraction requires the runtime capability set selection, number, inherited_define, and freq.

If inspection does not provide them, do not work around the gate in a profile. The target runtime is not qualified for this extraction path.

cmpcode is different: the implementation has a compatibility fallback only when the selection entity equals the identity scope. Otherwise cmpcode-free extraction is rejected rather than inventing geography.

Failure: schema/profile mismatch

Profile compilation fails when its hierarchy or variables cannot be reconciled with runtime inspection—for example, an unknown entity, a non-selectable selection entity, an invalid ancestor relationship, an ambiguous variable not explicitly blocked, or a parent map that contradicts runtime evidence.

Treat this as an adapter/schema problem. Re-inspect the database and compare the profile with the actual source. Do not change generic extraction logic merely to silence a database-specific mismatch.

Failure: extraction/normalization invariant

Errors about duplicate IDs, null structural dimensions, non-unit counts, incompatible batch key sets, or invalid canonical sequences indicate that the observed runtime result violates the record reconstruction contract.

Preserve the failing selection code, runtime versions/capabilities, profile hash, and generated evidence. Reproduce on the smallest affected partition before changing the model.

These failures are stronger than a downstream validation warning: the extractor refuses to construct a trusted relational result from the invalid intermediate state.

Failure: validation report says fail

Inspect the named checks in validation.json.

A failed count:<ENTITY> means the reconstructed row count disagrees with an explicit expected count. A failed unique:<key> means missing or duplicate declared primary keys. A failed fk:<child>-><parent> means one or more child keys are null or absent from the parent key set.

For extract-many, a failed validation prevents a success checkpoint. The partition's validation.json remains useful failure evidence.

Do not edit the report or checkpoint to pass. Diagnose the source/profile/runtime/contract and rerun.

Failure: output artifact was modified or corrupted

On the next resumable extract-many run, a mismatched Parquet hash or dataset-manifest semantic hash makes the old checkpoint non-reusable. The partition is recomputed.

This is the preferred recovery mechanism. You do not need to delete a valid-looking checkpoint by hand.

Standalone rxdb validate <dir> is not an artifact-integrity checker; it reads the persisted validation report. Use the resume path when you need checkpoint-bound artifact verification.

Failure during recomputation

Before attempting a non-skipped partition, orchestration invalidates the old success checkpoint. If the new extraction or validation then fails, the stale checkpoint is not left behind as evidence of success.

Some slice artifacts or a failing validation.json can still exist and are useful for diagnosis. The absence of a success checkpoint is intentional.

Interpreting run-manifest.json

A successful extract-many invocation writes a root run-manifest.json after partition processing. Use it to see which requested partitions were completed versus skipped and which hashes/runtime mode governed the run.

If orchestration aborts partway through, the root may still contain a manifest from an earlier successful invocation. Do not use its mere presence as proof of the failed attempt. Check timestamps/context externally and inspect current partition outputs/checkpoints.

Safe recovery sequence

For most operational failures:

1. keep the failing evidence
2. identify the smallest affected partition
3. inspect runtime/schema/profile inputs
4. reproduce with workers=1
5. use one-shot bridge if isolating persistent transport
6. fix the actual input/runtime/profile/code issue
7. rerun with normal resume
8. confirm the affected partition is completed and validation passes
9. only then scale concurrency/coverage again

This sequence lets the checkpoint contract decide what can be safely reused.

When local qualification is still required

Repository tests can establish the Python-side semantics documented above, but they cannot prove that a particular locally installed RedEngine build, RXDB corpus, filesystem, or memory envelope behaves correctly.

Before a new runtime/database combination is promoted to production-scale use, qualify it locally with:

  • inspect and captured capability/runtime versions;
  • one small known partition;
  • one representative larger partition;
  • validation evidence;
  • persistent-bridge smoke if that transport will be used;
  • a conservative worker/memory observation if concurrency will be used.

Those results belong in the database-specific qualification evidence, not as universal claims in this generic operations guide.

For artifact details see Outputs; for the meaning of a pass/fail report see Validation.