Get the data

Read OmicIDX Parquet from DuckDB, Python, or R over anonymous HTTPS.

Every OmicIDX table is a Parquet file under https://omicidx.cancerdatasci.org/data/. Anything that reads Parquet over HTTPS can use it. You don’t need an account or a key. There’s no REST API.

• DuckDB: Query with DuckDB covers the published database, every table, and example queries.
• R: the omicidxr package connects in one line and replaces SRAdb and GEOmetadb.
• In the browser: the playground runs SQL with DuckDB-Wasm, nothing to install.
• Pinning and citing: Versions and citation explains latest/, dated folders, and the monthly HuggingFace releases. Releases lists what’s available.

The recipes below use latest/ and ran when this page was built. Swap in v{YYYY-MM-DD}/ to pin a date.

Python

With polars, scan_parquet reads lazily, so only the columns and row groups a query needs are fetched. To try it without a project, run uv run --with polars python.

import polars as pl

base = "https://omicidx.cancerdatasci.org/data/latest"

studies = (
    pl.scan_parquet(f"{base}/sradb/study.parquet")
    .filter(pl.col("study_type") == "Transcriptome Analysis")
    .select("study_accession", "study_title")
    .head(10)
    .collect()
)
studies

shape: (10, 2)

study_accession	study_title
str	str
"DRP000003"	"Comprehensive identification a…
"DRP000004"	"Comprehensive identification a…
"DRP000005"	"Comprehensive identification a…
"DRP000006"	"Comprehensive identification a…
"DRP000007"	"Comprehensive identification a…
"DRP000008"	"Comprehensive identification a…
"DRP000011"	"Comprehensive analysis of cyto…
"DRP000012"	"Comprehensive analysis of cyto…
"DRP000013"	"Comprehensive analysis of cyto…
"DRP000014"	"Comprehensive analysis of poly…

With DuckDB, attach the published database and get a polars frame back. .pl() needs pyarrow installed, so run it with uv run --with duckdb --with polars --with pyarrow python:

import duckdb

con = duckdb.connect()
con.sql("ATTACH 'https://omicidx.cancerdatasci.org/data/latest/omicidx.duckdb' AS omicidx (READ_ONLY)")
gsms = con.sql("SELECT gsm FROM omicidx.geometadb.gse_gsm WHERE gse = 'GSE118849'").pl()
gsms.head()

shape: (5, 1)

gsm
str
"GSM3349112"
"GSM3349115"
"GSM3349119"
"GSM3349120"
"GSM3349118"

R

The omicidxr package connects for you, pins a version, and has SRAdb- and GEOmetadb-style helpers:

library(omicidxr)

con <- omicidx_connect()
gsms <- geo_samples(con, "GSE118849")
head(gsms[, c("gsm", "title", "gpl")])

gsm	title	gpl
GSM3349109	Exp #1-BAT-C	GPL17021
GSM3349110	Exp #2-BAT-C	GPL17021
GSM3349111	Exp #3-BAT-C	GPL17021
GSM3349112	Exp #1-BAT-TN	GPL17021
GSM3349113	Exp #2-BAT-TN	GPL17021
GSM3349114	Exp #3-BAT-TN	GPL17021

DBI::dbDisconnect(con)

Without the package, duckdb gives you the same database through DBI:

library(DBI)

con <- dbConnect(duckdb::duckdb())
dbExecute(con, "INSTALL httpfs; LOAD httpfs;")

#> [1] 0

dbExecute(con, "ATTACH 'https://omicidx.cancerdatasci.org/data/latest/omicidx.duckdb' AS omicidx (READ_ONLY)")

#> [1] 0

gsms <- dbGetQuery(con, "SELECT gsm FROM omicidx.geometadb.gse_gsm WHERE gse = 'GSE118849'")

# Any table, without the database
dbGetQuery(con, "
  SELECT count(*) AS n
  FROM read_parquet('https://omicidx.cancerdatasci.org/data/latest/sra_studies.parquet')
")

n
775372

dbDisconnect(con, shutdown = TRUE)

Command line

# Row counts for every published table
curl -s https://omicidx.cancerdatasci.org/data/latest/manifest.json

# One table, downloaded
curl -O https://omicidx.cancerdatasci.org/data/latest/geometadb/gse.parquet