smftools.informatics.ragged_store

Read-relative molecule storage and reference-grid materialization.

Ragged records keep per-base arrays in query coordinates. CIGAR placement is performed only when a dense reference-grid slice is requested, avoiding the mandatory padded matrices produced by the legacy load path.

Functions

alignment_to_ragged_record(read, ...[, ...])	Convert a pysam-like aligned segment to one read-relative record.
cigar_max_indel_runs(cigar)	Return (max_insertion_run, max_deletion_run) for a CIGAR string.
cigar_query_length(cigar)	Return the number of query bases consumed by cigar.
cigar_reference_length(cigar)	Return the number of reference positions consumed by cigar.
iter_cigar_aligned_pairs(cigar, reference_start)	Yield (query_position, reference_position) for aligned CIGAR bases.
materialize_ragged(frame, *, obs, ...[, ...])	Scatter selected read-relative records onto a dense reference grid.
materialize_ragged_streaming(chunks, *, obs, ...)	Memory-bounded materialize_ragged that consumes ragged frames in chunks.
parse_cigar(cigar)	Parse a SAM CIGAR string and reject malformed or unsupported input.
read_ragged_parquet(paths, *[, read_ids])	Read ragged parquet shard(s), optionally retaining selected read IDs.
strand_switch_metrics(signs)	Summarize an ordered deamination-vote track for PCR-chimera detection.
validate_ragged_frame(frame)	Validate and normalize a ragged molecule DataFrame.
write_ragged_parquet(frame, path)	Validate and write one shard of read-relative molecule records.

smftools.informatics.ragged_store.parse_cigar(cigar)

Parse a SAM CIGAR string and reject malformed or unsupported input.

Return type:

tuple[tuple[int, str], ...]

smftools.informatics.ragged_store.cigar_query_length(cigar)

Return the number of query bases consumed by cigar.

Return type:

int

smftools.informatics.ragged_store.cigar_reference_length(cigar)

Return the number of reference positions consumed by cigar.

Return type:

int

smftools.informatics.ragged_store.cigar_max_indel_runs(cigar)

Return (max_insertion_run, max_deletion_run) for a CIGAR string.

Each value is the length of the single longest internal insertion (I) or deletion (D) operation. Insertions and deletions in a SAM CIGAR are always internal to the aligned span (read ends are represented as soft/hard clips), so no edge trimming is required. Returns (0, 0) for an unmapped/absent CIGAR ("*" or empty) rather than raising.

Return type:

tuple[int, int]

smftools.informatics.ragged_store.strand_switch_metrics(signs)

Summarize an ordered deamination-vote track for PCR-chimera detection.

signs is a reference-ordered list of per-position strand votes, +1 for a C->T (top-consistent) event and -1 for a G->A (bottom-consistent) event. A pure molecule is all one sign; a PCR chimera is a run of one sign followed by a run of the other.

Returns (segment_purity, switch_index) where: :rtype: tuple[float, int]

• segment_purity is the best two-segment purity over all split points, i.e. (max(#+, #-) left + max(#+, #-) right) / n. It is ~1.0 for both a pure molecule and a clean chimera, and lower for a randomly interspersed (noisy) read. With fewer than two votes it is 1.0 (trivially consistent).

• switch_index is the index k (split before position k) of the best split whose left/right majorities have opposite sign, or -1 if no split separates opposite majorities. Callers map k to a reference coordinate.

smftools.informatics.ragged_store.iter_cigar_aligned_pairs(cigar, reference_start)

Yield (query_position, reference_position) for aligned CIGAR bases.

Return type:

Iterator[tuple[int, int]]

smftools.informatics.ragged_store.validate_ragged_frame(frame)

Validate and normalize a ragged molecule DataFrame.

Returns a copy so callers never have their input DataFrame mutated.

Return type:

DataFrame

smftools.informatics.ragged_store.write_ragged_parquet(frame, path)

Validate and write one shard of read-relative molecule records.

Return type:

Path

smftools.informatics.ragged_store.read_ragged_parquet(paths, *, read_ids=None)

Read ragged parquet shard(s), optionally retaining selected read IDs.

read_ids, when given, is pushed down as a pyarrow filter (pd.read_parquet(..., filters=[(read_id, "in", ...)])) rather than reading the whole shard into pandas and filtering afterward -- only the selected rows' ragged arrays are ever materialized, which is most of the remaining memory floor after streaming materialization (see dev/pipeline_scaling_audit.md, and _narrow_ragged_values above for the other half).

Return type:

DataFrame

smftools.informatics.ragged_store.alignment_to_ragged_record(read, reference_sequence, *, reference=None, reference_strand=None)

Convert a pysam-like aligned segment to one read-relative record.

Return type:

dict[str, object]

smftools.informatics.ragged_store.materialize_ragged(frame, *, obs, reference_lengths, layers=None, uns=None, start=None, end=None)

Scatter selected read-relative records onto a dense reference grid.

Whole-frame entry point: the caller already holds every selected read's ragged arrays in frame. materialize_ragged_streaming is the memory-bounded variant for large selections (it never holds more than one shard's ragged frame plus the dense output at once); both share _scatter_read_row so they cannot diverge.

Return type:

AnnData

smftools.informatics.ragged_store.materialize_ragged_streaming(chunks, *, obs, reference_lengths, layers=None, uns=None, start=None, end=None)

Memory-bounded materialize_ragged that consumes ragged frames in chunks.

obs (one row per selected read, in output order) fixes the dense output shape/order and, via its Reference_strand column, the reference lengths -- so the dense grids are preallocated once, up front, from obs alone. Each chunk (e.g. one parquet shard filtered to the selection) is then scattered into those grids and freed before the next arrives; peak memory is therefore ~one chunk's ragged frame + the dense output, independent of the total selection size. This is the fix for a real ~27x memory balloon: reading a whole selection's ragged arrays into one pandas frame before compacting to dense (int64/float64 list-columns are ~25-50x their dense footprint) drove a nominally-0.5GB preprocess task to 16-44GB peak (see dev/pipeline_scaling_audit.md). Reads in a chunk that are not in obs are ignored (a shard holds many barcodes' reads); a chunk with no matching reads contributes nothing.

Return type:

AnnData