smftools.informatics.raw_store

Write the ragged source-of-truth store and its thin molecule spine.

Functions

write_raw_store(frame, output_dir, *, ...[, ...])	Write ragged parquet shards, a thin spine, and manifest registration.
write_raw_store_streaming(reference_groups, ...)	Streaming entry point: consumes bounded-size, possibly-repeated-per- reference groups (see _write_raw_shards_streaming), never holding more than one incoming group's ragged data (sequence/quality/ mismatch/modification_signal) in memory at once.

smftools.informatics.raw_store.write_raw_store(frame, output_dir, *, reference_lengths, shard_size=100000, start_bin_size=1000000, analysis_mode='auto', load_cache_mode='auto', max_full_matrix_gb=8.0, genome_tile_size=10000, genome_tile_halo=1000, bam_path=None, extra_uns=None)

Write ragged parquet shards, a thin spine, and manifest registration.

Parameters:

• frame (DataFrame) -- One row per physical read using the ragged schema.

• output_dir (str | Path) -- Experiment load output directory.

• reference_lengths (Mapping[str, int]) -- Exact lengths keyed by Reference_strand.

• shard_size (int (default: 100000)) -- Maximum reads per parquet shard.

• start_bin_size (int (default: 1000000)) -- Width used to group reads by alignment start.

• bam_path (str | Path | None (default: None)) -- Optional aligned BAM pointer copied to every spine row.

• extra_uns (Optional[Mapping[str, object]] (default: None)) -- Additional self-describing metadata for the spine.

Return type:

dict[str, object]

Returns:

Dictionary containing spine, ragged_store, and manifest paths.

Whole-frame convenience entry point: the caller already has every read in memory (tests, or any other caller assembling a frame directly), so this provides no memory-scaling benefit on its own -- cli/raw_adata.py:: build_ragged_records is the streaming producer that gets the actual win, by never materializing more than one reference's frame at a time before calling _write_raw_shards_streaming directly. This function groups by Reference_strand and feeds the same streaming core, so its output (shard contents, catalog, molecules, barcode index) is unchanged from before this split -- see that function's docstring for why.

smftools.informatics.raw_store.write_raw_store_streaming(reference_groups, output_dir, *, reference_lengths, shard_size=100000, start_bin_size=1000000, analysis_mode='auto', load_cache_mode='auto', max_full_matrix_gb=8.0, genome_tile_size=10000, genome_tile_halo=1000, bam_path=None, extra_uns=None)

Streaming entry point: consumes bounded-size, possibly-repeated-per- reference groups (see _write_raw_shards_streaming), never holding more than one incoming group's ragged data (sequence/quality/ mismatch/modification_signal) in memory at once. cli/raw_adata.py::build_ragged_records_streaming is the intended producer -- for a low-read-depth reference it may still extract and attach one reference's rows as a single group, but for a high-read-depth one it flushes that reference's data across several bounded-size groups instead of accumulating the whole thing first (see dev/pipeline_scaling_audit.md).

Every other artifact this writes (catalog, molecules, barcode index, spine obs/uns, manifest) is identical in shape to write_raw_store's -- this function differs only in how much of the experiment it needs resident in memory to produce them. Each group's obs contribution is built and appended immediately (safe regardless of how many groups a reference is split into -- a read's row never repeats across groups), but plan_references needs a reference's true total read count to choose a correct locus/genome plan, so it can't be computed from a partial group. This function accumulates a per-reference row count (a handful of integers, not ragged data) across a reference's groups and only calls plan_references -- using a synthetic single-column frame that reproduces the true final Reference_strand value_counts without ever holding the real ragged rows -- once is_final is True for that reference.

Each yielded frame must already be validated (validate_ragged_frame) by the caller -- unlike write_raw_store, this function never sees the whole experiment at once, so it cannot run that check across references itself; only within-reference invariants are enforced downstream.

Return type:

dict[str, object]