Architecture

How NCBI feeds become the public Parquet files. Scheduled extracts, an internal DuckLake, SQLMesh marts, and a nightly export.

OmicIDX is a batch pipeline. Every job is a plain Python process run by a systemd timer. There’s no orchestrator and no database serving the public.

NCBI FTP / E-utilities
        │   per-source extracts, one systemd timer each
        ▼
raw Parquet / NDJSON on Cloudflare R2
        │   nightly: MERGE into the lake
        ▼
internal DuckLake (Postgres catalog, data on R2)
        │   SQLMesh: staging → sradb / geometadb marts
        ▼
nightly Parquet export ──▶ omicidx.cancerdatasci.org/data/
                              latest/, v{date}/
                                │ monthly
                                ▼
                         HuggingFace seandavis/omicidx (tagged, DOI)

Stages

  1. Extract. One job per source (SRA, GEO, BioSample/BioProject, PubMed, and EBI BioSamples) downloads new upstream data and writes raw files to R2. A small marker file records each finished partition, so a failed job resumes where it stopped. Parsing uses omicidx-parsers.
  2. Load. Every night, the raw files are merged into an internal DuckLake (Postgres catalog, Parquet data on R2). The lake keeps snapshots, so earlier states can be reproduced.
  3. Transform. SQLMesh builds the sradb and geometadb marts from the lake tables.
  4. Publish. The base tables and marts are exported to public Parquet: a new v{date}/ folder, a refreshed latest/, an omicidx.duckdb of views, views.sql, restore.sql (a local-copy script), and manifest.json. A monthly job copies one release to HuggingFace.

Extraction and publishing are decoupled. The nightly build uses whatever raw data is already on R2, so a slow or failing source only delays its own tables.

Provenance

Each release’s manifest.json records the publish date, the git commit of the transform code (transform_sha), the internal lake snapshot it was exported from (lake_snapshot_id), and per-table row counts. See Versions and citation.

Source code

Everything is in github.com/seandavi/omicidx: parsers in packages/omicidx-parsers, the pipeline in packages/omicidx-prefect, and the systemd units in systemd/.