smftools documentation

smftools documentation#

smftools#

A Python tool for automated processing of raw sequencing data derived from single molecule footprinting experiments into zarr and parquet data formats. Experimental data is organized into projects using DuckDB (optional; falls back to a pandas/pyarrow union without it) to facilitate growth of data collections for scientific projects. An additional analysis subpackage provides functionality that can be imported for custom analyses and interactive analysis sessions.

Philosophy#

While genomic data structures (SAM/BAM) were built to store read alignment data and basic read metadata along large references, integration of downstream analyses is not feasible using this format alone. Smftools integrates experimental analyses across file formats, linking raw sequencing data files, BAM alignment files, and downstream analyses into modern storage formats such as zarr for arrays and parquet for tables. This enables efficient partitioned data storage, rapid and parallel data operations, hierarchical metadata handling, and seamless integration with machine-learning workflows. Furthermore, functionality is modularized into multiple processing stages, enabling analysis to restart from convenient checkpoints without having to rerun the full workflow. Collections of experiments are managed under smftools projects, which indexes individual experiments and combine them for continuously growing scientific projects.

Installation#

PyPI releases lag behind active development (currently 0.3.2 on PyPI vs. the version in this repo), so installing from source is recommended for now:

git clone https://github.com/jkmckenna/smftools.git
cd smftools
pip install -e ".[torch]"      # core install; torch is listed as an extra but is currently required
pip install -e ".[all]"        # everything, including optional extras below

Optional extras can be installed individually as needed, e.g. pip install -e ".[ont,plotting]":

ont -> POD5 I/O for Nanopore data.
plotting -> matplotlib/seaborn-based figure generation.
ml-base / ml-extended -> scikit-learn/torch-based and extended (captum, lightning, wandb, ...) machine-learning workflows.
umap -> UMAP-based dimensionality reduction.
cluster -> igraph/leidenalg-based clustering.
catalog -> DuckDB-backed project catalog (falls back to pandas/pyarrow without it).
qc -> MultiQC report generation.

See pyproject.toml's [project.optional-dependencies] for the full, current list.

Command-line interface#

smftools exposes two top-level command groups (smftools --help for the full list):

smftools experiment <config_path> -> pipeline stages for a single experiment:

raw -> Prepare BAM artifacts and write the ragged raw store.
load -> Optionally pre-build the dense zarr cache from raw artifacts.
preprocess -> QC, filtering, and read-level preprocessing.
spatial -> Spatial signal analysis.
hmm -> HMM feature annotation and plotting.
full -> Composed workflow: raw, preprocess, spatial, hmm.
batch -> Run any single stage across many experiments listed in a CSV/TSV/TXT file.
concatenate, export-fastq, plot-current -> supporting utilities.

smftools project <project_dir> -> registering and querying across experiments:

init -> Initialize a project directory + registry.
add / remove -> Register or deactivate an experiment in the project.
list -> List registered experiments and harmonized references.
materialize -> Pool a reference across matching experiments into one AnnData.
sample-store-list -> List cataloged per-sample-store partitions.
export-fastq -> Write one FASTQ per barcode of QC-passed reads, across every registered experiment.