Data sources

Where each OmicIDX table comes from, how often it refreshes, and which public file it lands in.

Each source is extracted on its own schedule. A nightly build then publishes whatever has been extracted so far, so one slow source never holds up the rest.

Source	Upstream	Extract cadence	Public files
NCBI SRA	SRA mirroring XML, plus SRA_Accessions.tab	daily	sra_studies, sra_samples, sra_experiments, sra_runs, sra_accessions; sradb/*
NCBI GEO	GEO SOFT records via E-utilities and acc.cgi, one calendar month at a time	daily (current month re-fetched)	geo_series, geo_samples, geo_platforms, geo_series_with_rnaseq_counts; geometadb/*
NCBI BioSample	biosample_set.xml.gz full dump	daily	biosamples
NCBI BioProject	bioproject.xml full dump	daily	bioprojects
NCBI PubMed	PubMed baseline and update files	hourly check for new files	pubmed_articles
NCBI Taxonomy	taxdump.tar.gz	weekly	none: used to resolve organism names

Every table is one Parquet file: <name>.parquet at the top of a release, or sradb/<name>.parquet / geometadb/<name>.parquet for the compatibility tables.

Notes per source

• SRA: NCBI publishes full and incremental mirror files. OmicIDX loads them in order and keeps the latest version of each record. sra_accessions is NCBI’s accession table (status, dates, relationships for every SRA accession).
• GEO: records are fetched one month at a time, by last-update date. The current month is re-fetched every night because it keeps growing. geo_series_with_rnaseq_counts lists the series for which NCBI provides computed RNA-seq count matrices.
• BioSample and BioProject: NCBI publishes these as single full dumps, so each night replaces the previous load.
• PubMed: the annual baseline plus the daily update files. New files are picked up within the hour and published in the next nightly build. pubmed_articles.parquet is large (about 37 GB), so filter it in place rather than downloading it.
• Organism names: NCBI Taxonomy renames organisms, and GEO updates its records to match without changing their last-update date. Organism names in geo_series, geo_samples, sra_samples, biosamples and the matching sradb/ and geometadb/ tables are therefore the current NCBI Taxonomy scientific name for the record’s taxonomy ID (merged IDs are followed). The name as originally submitted is kept in a matching *_as_submitted column (for example sample_organism_as_submitted). If a taxonomy ID is unknown, the submitted name is used. geo_platforms has no taxonomy ID, so platform organisms stay as submitted.
• EBI BioSamples: extracted daily into the internal lake (by update date, from 2021-01-01), but not published yet.

Compatibility tables

• sradb/ (study, sample, experiment, run, sra, run_with_study, rnaseq_runs, wgs_runs, human_runs, mouse_runs): SRAdb-style names and columns. Legacy columns that are always empty are dropped. Modern cross-references (BioProject, GEO, PubMed) are added.
• geometadb/ (gse, gsm, gpl, gse_gsm, gse_gpl, geo_supplemental_files): GEOmetadb-style tables.

Row counts for every published table are in each release’s manifest.json.