smftools.informatics.partition_store

smftools.informatics.partition_store#

Zarr partition + thin molecule-index spine + partition catalog helpers.

write_dense_cache_from_spine is the actively-used piece: it builds the optional dense zarr cache (smftools load / cli.load_adata.load_dense_cache) from a raw ragged store, and is what AdataPaths.store/.spine/.catalog point at.

The lower-level write_partitioned_store/build_spine/write_catalog/ write_experiment_store functions predate the raw/load split (they were the storage layer for the original 1.0.0 output re-architecture spike, partitioning an arbitrary in-memory AnnData by (Reference_strand, Sample) with no genomic windowing). Nothing in the current pipeline writes that format anymore -- the CLI command and load_adata_core auto-emit that used to produce it have been removed. They're kept only because materialize()'s generic dense-partition read path (tests/unit/informatics/test_partition_read.py, test_partition_store.py) still uses them as a convenient way to build an arbitrary multi-reference/multi-sample dense fixture without needing a full ragged/CIGAR-based raw store. Do not wire them back into production code; if that read path needs new coverage, build it from write_dense_cache_from_spine or write_raw_store instead.

The partition + spine writes go through smftools.readwrite.safe_write_zarr() / safe_write_h5ad(), so the same sanitization + pickle-backup behavior used by the monolithic path applies here too. Partitions are written zarr v3.

Functions

build_spine(adata, partitions, *[, ...])

Build the thin molecule-index spine AnnData from written partitions.

write_catalog(catalog_path, partitions, *[, ...])

Write the partition catalog parquet (one row per partition).

write_dense_cache_from_spine(spine_path, *)

Materialize and persist a reference-partitioned dense cache.

write_experiment_store(adata, output_dir, *)

Write a full experiment store: partitions + spine + catalog, and register them.

write_partitioned_store(adata, output_dir, *)

Split adata by (reference, sample) and write one zarr partition each.

Classes

PartitionInfo(partition_id, reference, ...)

Metadata for one written (reference, sample) partition.
class smftools.informatics.partition_store.PartitionInfo(partition_id, reference, sample, group_path, n_reads, n_positions, read_ids=<factory>)#

Bases: object

Metadata for one written (reference, sample) partition.

Fields:

partition_id: Stable identifier ("<slug_ref>/<slug_sample>"). reference: Exact Reference_strand value for this partition. sample: Exact Sample value for this partition. group_path: Partition zarr path, relative to the experiment output dir. n_reads: Number of molecules (obs rows) in the partition. n_positions: Number of positions (var) in the partition. read_ids: Read names in stored order (index i -> partition_row i).

partition_id: str#
reference: str#
sample: str#
group_path: str#
n_reads: int#
n_positions: int#
read_ids: list[str]#
smftools.informatics.partition_store.write_partitioned_store(adata, output_dir, *, reference_col='Reference_strand', sample_col='Sample', obs_chunk=10000, zarr_format=3, backup=True, verbose=False)#

Split adata by (reference, sample) and write one zarr partition each.

Parameters:

  • adata (AnnData) -- In-memory AnnData to partition (X + layers + obs + var + uns).

  • output_dir (str | Path) -- Experiment output directory; partitions go under output_dir/store/<slug_ref>/<slug_sample>.

  • reference_col (str (default: 'Reference_strand')) -- obs column identifying the reference (default Reference_strand).

  • sample_col (str (default: 'Sample')) -- obs column identifying the sample (default Sample).

  • obs_chunk (int (default: 10000)) -- zarr chunk size along the obs axis.

  • zarr_format (int (default: 3)) -- zarr on-disk format to write (default v3).

  • backup (bool (default: True)) -- Passed to safe_write_zarr (pickle non-serializable uns/obs).

  • verbose (bool (default: False)) -- Verbose sanitization logging.

Returns:

One entry per written partition, with read order.

Return type:

list[PartitionInfo]

Raises:

KeyError -- If reference_col or sample_col is missing from adata.obs.

smftools.informatics.partition_store.build_spine(adata, partitions, *, identity_cols=None, bam_path=None, extra_uns=None)#

Build the thin molecule-index spine AnnData from written partitions.

One obs row per molecule, indexed by read name, with identity columns plus pointer columns (partition group path, partition_row position within that partition, and bam_path). Carries no X or layers.

Parameters:

  • adata (AnnData) -- The source AnnData (for identity columns and uns pointers).

  • partitions (Sequence[PartitionInfo]) -- Partition metadata from write_partitioned_store().

  • identity_cols (Optional[Sequence[str]] (default: None)) -- obs columns to copy as identity; defaults to those in DEFAULT_IDENTITY_COLS that exist on adata.

  • bam_path (str | None (default: None)) -- Optional aligned-BAM path recorded for every molecule.

  • extra_uns (dict | None (default: None)) -- Extra key/values to store in the spine uns.

Returns:

The spine (obs-only) object.

Return type:

anndata.AnnData

smftools.informatics.partition_store.write_catalog(catalog_path, partitions, *, experiment=None, modality=None, config_hash=None)#

Write the partition catalog parquet (one row per partition).

Parameters:

  • catalog_path (str | Path) -- Output parquet path.

  • partitions (Sequence[PartitionInfo]) -- Partition metadata from write_partitioned_store().

  • experiment (str | None (default: None)) -- Optional experiment name recorded on every row.

  • modality (str | None (default: None)) -- Optional smf modality recorded on every row.

  • config_hash (str | None (default: None)) -- Optional config hash recorded on every row (staleness check).

Returns:

The written catalog path.

Return type:

Path

smftools.informatics.partition_store.write_experiment_store(adata, output_dir, *, experiment=None, modality=None, config_hash=None, bam_path=None, reference_col='Reference_strand', sample_col='Sample', obs_chunk=10000, zarr_format=3, verbose=False)#

Write a full experiment store: partitions + spine + catalog, and register them.

Parameters:

  • adata (AnnData) -- The in-memory experiment AnnData to convert.

  • output_dir (str | Path) -- Experiment output directory.

  • experiment (str | None (default: None)) -- Experiment name (recorded in catalog and spine uns).

  • modality (str | None (default: None)) -- smf modality (recorded in catalog).

  • config_hash (str | None (default: None)) -- Config hash (recorded in catalog for staleness detection).

  • bam_path (str | None (default: None)) -- Aligned-BAM path recorded per molecule in the spine.

  • reference_col (str (default: 'Reference_strand')) -- obs column identifying the reference.

  • sample_col (str (default: 'Sample')) -- obs column identifying the sample.

  • obs_chunk (int (default: 10000)) -- zarr obs-axis chunk size.

  • zarr_format (int (default: 3)) -- zarr on-disk format (default v3).

  • verbose (bool (default: False)) -- Verbose sanitization logging.

Returns:

Paths for store, spine, catalog, manifest.

Return type:

dict[str, Path]

smftools.informatics.partition_store.write_dense_cache_from_spine(spine_path, *, output_dir=None, obs_chunk=10000, zarr_format=3, verbose=False)#

Materialize and persist a reference-partitioned dense cache.

Each reference is densified independently from the ragged store, ordered by barcode, and written before the next reference is loaded. This bounds peak memory by the largest reference partition and preserves one shared row layout.

Return type:

dict[str, Path]