Versions and citation

latest vs dated vs monthly releases, how to pin a version, and how to cite OmicIDX.

OmicIDX publishes three kinds of versions. They hold the same tables in the same layout; they differ in how long they last and how you use them. Releases lists the ones available now.

Version Where Changes? Kept Use it by
latest/ https://omicidx.cancerdatasci.org/data/latest/ rebuilt every night always the newest querying in place
Dated https://omicidx.cancerdatasci.org/data/v{YYYY-MM-DD}/ never 30 days querying in place
Monthly release HuggingFace dataset seandavis/omicidx, git tag YYYY-MM never long term, with a DOI downloading, then querying locally

latest/ and the dated folders are served for querying in place: DuckDB ATTACH, read_parquet, omicidxr, and the playground all read them over HTTPS. The monthly releases are the long-term archive and the version to cite. Download one, then query it locally.

Which one to use

  • Exploring, or building something that should track NCBI: use latest/.
  • An analysis you’ll rerun in the next few weeks: pin a dated folder.
  • A paper, a shared pipeline, or anything you’ll cite: download a monthly release.

Pin a dated folder

Replace latest/ with a date:

SELECT count(*)
FROM read_parquet('https://omicidx.cancerdatasci.org/data/v2026-10-20/sradb/study.parquet');

To see which dates are available, open Releases or the folder listing. Dated folders older than 30 days are deleted. Use a monthly release for anything longer.

Caution

In dated folders published before late October 2026, omicidx.duckdb and views.sql point at latest/, not at their own folder. For those dates, pin with read_parquet on the dated URLs, as above.

Use a monthly release

Once a month, one nightly build is copied to the HuggingFace dataset seandavis/omicidx and tagged with its month (e.g. 2026-10). Files sit at the repository root in the same layout as R2 (sra_runs.parquet, sradb/study.parquet, …).

Download the release with the hf command-line tool (in huggingface_hub; uvx --from huggingface_hub hf runs it without installing), then query the local files:

hf download seandavis/omicidx --repo-type dataset --revision 2026-10 \
  --local-dir omicidx-2026-10
duckdb -c "SELECT count(*) FROM 'omicidx-2026-10/sradb/study.parquet'"

--include 'sradb/*' downloads only some of the files. A HuggingFace token (hf auth login) raises the download rate limits.

In R, omicidx_download("2026-10", "omicidx-2026-10.duckdb") builds a local DuckDB database from a monthly release and checks its row counts (see omicidxr). For latest/ and dated folders, restore.sql does the same from the command line.

DuckDB can also read a release in place with hf:// paths, where @tag pins the release:

SELECT count(*)
FROM 'hf://datasets/seandavis/omicidx@2026-10/sradb/study.parquet';

That’s fine for a quick look, but HuggingFace limits anonymous requests to 3000 per 5 minutes per IP address, and a query can make hundreds, so repeated queries soon fail with HTTP 429. Download for real work.

Cite OmicIDX

Each monthly release gets a DOI, minted on HuggingFace. Cite the DOI of the release you used, which you’ll find on the dataset page for that tag. In your methods, also state the release tag (e.g. 2026-10).

If you used a dated folder instead, record its URL and the transform_sha and lake_snapshot_id from its manifest.json. The folder will be gone in 30 days, so a monthly release is the better thing to cite.

The manifest

Every version has a manifest.json:

curl -s https://omicidx.cancerdatasci.org/data/latest/manifest.json

DuckDB reads it directly. The provenance fields of the current latest/:

SELECT publish_date, dated_path, transform_sha, lake_snapshot_id
FROM read_json('https://omicidx.cancerdatasci.org/data/latest/manifest.json');
1 records
publish_date dated_path transform_sha lake_snapshot_id
2026-10-23 v2026-10-23/ 2a999e937697b61bfdb3294de36582dd1cc5ab1f 4239

And the row count of each published table:

SELECT t.schema, t."table", t.row_count
FROM read_json('https://omicidx.cancerdatasci.org/data/latest/manifest.json'),
     unnest(tables) AS u(t)
ORDER BY t.schema, t."table";
28 records
schema table row_count
geometadb geo_supplemental_files 7098064
geometadb gpl 29060
geometadb gse 298108
geometadb gse_gpl 345076
geometadb gse_gsm 10562000
geometadb gsm 8774090
main bioprojects 1130284
main biosamples 60484224
main geo_platforms 29060
main geo_samples 8774090
main geo_series 298108
main geo_series_with_rnaseq_counts 27714
main pubmed_articles 41275240
main sra_accessions 162991962
main sra_experiments 42655706
main sra_runs 45553970
main sra_samples 44670639
main sra_studies 775372
sradb experiment 42655706
sradb human_runs 8540117
sradb mouse_runs 4415567
sradb rnaseq_runs 9054381
sradb run 45699354
sradb run_with_study 45699354
sradb sample 44670639
sradb sra 45699354
sradb study 775372
sradb wgs_runs 9601400
  • transform_sha is the commit of the pipeline code that built the tables.
  • lake_snapshot_id is the snapshot of the internal lake the tables were exported from.
  • tables gives the row count of each published table. raw_partition_counts records how many upstream partitions each source had.