smftools.informatics.partition_read

smftools.informatics.partition_read#

Assemble a concrete AnnData selection through the thin molecule spine.

This is the Phase 2 (1.1.x) reader that lets downstream stages stop materializing the whole monolithic AnnData. A selection (by reference / sample / read_id / obs mask) is resolved against the spine's pointer columns, the referenced partitions are loaded (only those), subset to the selected reads, and concatenated.

Two backends:

  • eager (default, no extra deps): safe_read_zarr each needed partition.

  • lazy (lazy=True, needs xarray for anndata.experimental.read_lazy): dask-backed row selection then to_memory; falls back to eager if unavailable.

When the selected dense cache partitions do not exist, the reader falls back to the spine's read-relative parquet shards and performs CIGAR placement on demand. Either way, peak memory scales with the selection rather than the full dataset.

Functions

load_spine(spine_path, *[, verbose])

Load the thin molecule-index spine (e.g. written by write_raw_store, execute_partitioned_preprocessing, or another stage's spine writer).

materialize(spine, *[, base_dir, ...])

Assemble a concrete AnnData for a molecule selection from the partitions.

read_catalog(catalog_path)

Read a partition catalog.parquet into a DataFrame.

relative_uns_path(path, anchor)

Return path as a POSIX-style string relative to anchor, for storing on a spine.

resolve_relative_path(value, anchor)

Resolve a stored spine path value written by relative_uns_path().
smftools.informatics.partition_read.read_catalog(catalog_path)#

Read a partition catalog.parquet into a DataFrame.

Return type:

DataFrame

smftools.informatics.partition_read.load_spine(spine_path, *, verbose=True)#

Load the thin molecule-index spine (e.g. written by write_raw_store, execute_partitioned_preprocessing, or another stage's spine writer).

Return type:

AnnData

smftools.informatics.partition_read.relative_uns_path(path, anchor)#

Return path as a POSIX-style string relative to anchor, for storing on a spine.

Used instead of an absolute (.resolve()'d) string so cross-artifact pointers (e.g. a preprocess spine's pointer back to its source raw spine, or a raw spine's obs["bam_path"]) survive the containing directory tree being copied to a different machine or mount point. General-purpose despite the name -- applies equally to a uns value or a per-read obs column, anything that stores a path string on the spine. Pair with resolve_relative_path() on the read side.

For any value that might be inherited unchanged into a later stage's spine via spine.copy() (e.g. source_base_dir, preprocess_catalog, obs["bam_path"]), pass a stage-independent anchor -- see _run_root_from_spine_path() -- since a later stage's spine file lives in a different directory than the one that originally wrote the value, so an anchor tied to "wherever this spine currently lives" would resolve correctly there but not once copied elsewhere.

Return type:

str

smftools.informatics.partition_read.resolve_relative_path(value, anchor)#

Resolve a stored spine path value written by relative_uns_path().

Accepts both the new relative encoding (resolved against anchor) and the historical absolute-string encoding (used as-is, for old spines written before this fix), so already-written spines keep working without being regenerated. Returns None if value is absent, or relative with no anchor available.

Return type:

Path | None

smftools.informatics.partition_read.materialize(spine, *, base_dir=None, references=None, samples=None, read_ids=None, obs_mask=None, layers=None, read_metrics=False, lazy=False, start=None, end=None)#

Assemble a concrete AnnData for a molecule selection from the partitions.

Only the partitions referenced by the selection are read, so peak memory scales with the selection, not the whole dataset.

Parameters:

  • spine -- The spine AnnData or a path to spine.h5ad.

  • base_dir (str | Path | None (default: None)) -- Root for resolving partition group_path values. Defaults to the spine file's directory (when a path is given) or uns['store_root']'s parent (in-memory spine).

  • references (default: None) -- Optional Reference_strand value(s) to include.

  • samples (default: None) -- Optional Sample value(s) to include.

  • read_ids (default: None) -- Optional explicit read id(s) to include.

  • obs_mask (default: None) -- Optional boolean array aligned to spine.obs rows.

  • layers (Optional[Iterable[str]] (default: None)) -- Optional subset of layers to load (default: all).

  • read_metrics (Union[bool, Iterable[str]] (default: False)) -- Attach spatial-stage per-read outputs (autocorrelation, Lomb-Scargle periodograms, ...) if the spine has a spatial stage available -- True for everything found, a name subset, or False (default) to skip. Unlike layers, off by default: it's a different axis than genomic position (obs/obsm, not layers) and opening every matching task's read_metrics.zarr isn't free.

  • lazy (bool (default: False)) -- Use anndata.experimental.read_lazy (needs xarray); falls back to eager safe_read_zarr if unavailable.

  • start (int | None (default: None)) -- Optional zero-based inclusive genomic start.

  • end (int | None (default: None)) -- Optional zero-based exclusive genomic end. Must accompany start.

Returns:

The materialized selection, ordered as in the spine, with decoder/reference maps copied from the spine uns.

Return type:

anndata.AnnData

Raises:

ValueError -- If the selection matches no molecules or neither a complete dense cache nor ragged-store pointers are available.