Release notes

Release notes#

Version 2.14.0#

2.14.0 2026-07-16 Version 2.14.0 converted barcode-distribution box-and-whisker plots to violin plots, and fixed the HMM length-clustermap panel ignoring reordered column sites.

Version 2.13.0#

2.13.0 2026-07-16 Version 2.13.0 completed the signed-reindexing coordinate transform (Track 2): wired reindexing_offsets/reindexing_invert into the remaining absolute-coordinate call sites, including variant_adata.py's clustermap functions and rolling_nn_distance.py and its chimeric callers.

Version 2.12.0#

2.12.0 2026-07-16 Version 2.12.0 began the signed-reindexing coordinate transform (Track 1 of 2, continued in 2.13.0): per-reference sign-aware reindexing plus display-time column inversion, superseding the older invert_adata. Also documented reindexing_invert's already-working global-bool shorthand.

Version 2.11.0#

2.11.0 2026-07-16 Version 2.11.0 ported reindex_references_adata into the partitioned spatial/HMM clustermaps for invert/reindex parity with the older pipeline, added opt-in read-span cropping to spatial/HMM clustermap x-axes, and overlaid jittered per-read points on preprocess/spatial box-and-whisker plots.

Version 2.10.0#

2.10.0 2026-07-16 Version 2.10.0 fixed a raw-ingestion bug where Reference_strand could collide across conversion-state records.

Version 2.9.0#

2.9.0 2026-07-16 Version 2.9.0 fixed HMM feature layers to always be masked outside the read span (previously this only happened for multi-channel signals), and added per-barcode feature count/size histograms to HMM feature plots.

Version 2.8.0#

2.8.0 2026-07-15 Version 2.8.0 fixed a real correctness bug in HMM fitting: the EM log-likelihood was computed from the gamma sum instead of the forward-pass normalizer, causing early stopping to fire spuriously (as early as iteration 2). Also switched EM convergence to a relative tolerance (lowering the default to 1e-5), eliminated a redundant duplicate forward-backward pass in fit_em, defaulted HMM fitting/decoding to CPU instead of auto-detected GPU, and parallelized HMM clustermap plotting across cfg.threads.

Version 2.7.0#

2.7.0 2026-07-15 Version 2.7.0 added persistence of HMM EM fit history with per-checkpoint convergence plotting, and gated deaminase PCR chimeras out of read QC (with the QC/dedup chain documented).

Version 2.6.0#

2.6.0 2026-07-15 Version 2.6.0 added an HMM emission/transition parameter comparison plot across barcodes, for diagnosing per-barcode fit differences.

Version 2.5.0#

2.5.0 2026-07-15 Version 2.5.0 parallelized preprocess/spatial/hmm task execution using memory-aware worker pools.

Version 2.4.0#

2.4.0 2026-07-14 Version 2.4.0 added a pysam-based direct-modality MM/ML tag backend as an alternative to modkit, along with parallel raw-ingestion extraction.

Version 2.3.0#

2.3.0 2026-07-14 Version 2.3.0 continued the scaling audit: raw ingestion for the conversion/deaminase modalities is now streamed instead of building one full-experiment DataFrame in memory.

Version 2.2.0#

2.2.0 2026-07-14 Version 2.2.0 began the pipeline scaling audit (Tracks A-F, continued through 2.8.0): preprocess-derived layers are now streamed directly to zarr as they're computed, instead of being batched fully in memory first.

Version 2.1.0#

2.1.0 2026-07-14 Version 2.1.0 introduced the project-level catalog: a DuckDB-backed registry (project init/add/remove/list/materialize) that harmonizes references across experiments via a canonical, sequence-hash reference_uid computed at the raw stage, and pools matching data across experiments for cross-experiment materialize(). The CLI was stratified into experiment/project command groups. Added a project-level per-sample store built up in phases (legacy-experiment caching, composition-hash cache invalidation, set-level embedding persistence with transform-by-default, and a per-sample periodicity-analysis catalog), a consolidated experiment_spine.h5ad superset cache, normalized per-stage obs.parquet, an experiment_manifest.json provenance record, and a barcode-sorted raw store with molecules.parquet/barcode_index.parquet. Cross-machine path portability was addressed by making stage spine pointers, the project registry, and bam_path relative/portable. Also added deaminase PCR chimera detection (C->T / G->A strand-switch), a CIGAR-based internal-indel read filter, per-barcode chimera-rate QC plotting, partitioned HMM merged footprint plotting, and FASTQ export of QC-passed reads per barcode or per project.

Version 2.0.0#

2.0.0 2026-07-12 Version 2.0.0 split smftools experiment load into separate raw-ingestion and dense-cache stages, replacing the 1.x partitioned-store design. Raw ingestion became ragged and partitioned, reads are linked back to their origin POD5 file with per-read sequencing/signal metadata, and per-base current-signal features are derived from the move table into densifiable layers.

Version 1.1.0#

1.1.0 2026-07-12 Version 1.1.0 added a materialize() partition reader that lazily reads only the requested reference/sample slices from the partitioned store, rather than loading the full experiment.

Version 1.0.0#

1.0.0 2026-07-12 Version 1.0.0 introduced the partitioned-store architecture: experiment data is now written as zarr partitions (keyed by reference x sample) plus a thin spine.h5ad molecule index and a catalog, replacing the single monolithic AnnData file. Added zarr-safe I/O (reusing the h5ad sanitizer), made smftools experiment load additively emit the partitioned store, and added a migrate-store CLI command plus store paths on AdataPaths. This design was superseded in 2.0.0 by the raw/load split and in 2.1.0 by the project-level catalog, but established the on-disk partitioning model both build on.

Version 0.4.5#

0.4.5 2026-07-12 Version 0.4.5 removed a wasteful disk round-trip in smftools experiment load's sequence-encoding step: previously each batch of integer-encoded read sequences was written to a throwaway .h5ad file and immediately read back in the same worker process, with no memory benefit. Sequences are now encoded directly into an in-memory mapping, with a new unit test covering the encoding path.

Version 0.4.4#

0.4.4 2026-07-08 Version 0.4.4 fixed a crash in the direct-modification-detection modality and reduced load_adata cost for dual-modality runs.

Version 0.4.3#

0.4.3 2026-07-08 Version 0.4.3 focused on memory reduction in smftools load/preprocessing for large datasets, alongside periodicity-analysis updates.

Version 0.4.2#

0.4.2 2026-06-23 Version 0.4.2 introduced the smftools.analysis downstream analysis subpackage (with read caching, docstrings, and basic unit tests), simplified HMM peak calling, and parallelized preprocessing, plotting, and duplicate-flagging for large datasets. BAM QC gained streaming support, and an xgboost extra was added.

Version 0.4.1#

0.4.1 2026-02-23 Version 0.4.1 overhauled barcode/UMI handling: added UMI preprocessing and bipartite analysis, sidecar-based UMI extraction with a samtools backend, multiprocessing for barcode/UMI extraction, and a custom barcode demultiplexing backend (alongside dorado demux single/double-pass support and updated barcode kit definitions). Also added variant plotting broken out by UMI group, read-length preprocessing QC plots, a load_adata path manifest, and POD5 current-trace plotting.

Version 0.4.0#

0.4.0 2026-02-05 Version 0.4.0 was primarily a stabilization release, rolling up and finalizing the chimeric/variant work from 0.3.2 with minor preprocessing fixes.

Version 0.3.2#

0.3.2 2026-02-03 Version 0.3.2 introduced the chimeric and variant CLI pipeline stages with dedicated plotting, built out cross-sample PCR chimera detection on top of the zero-Hamming span work from 0.3.1 (merge-gap options, overlap modes, NaN run limits), and reorganized plotting code into workflow-specific modules. Latent-space plotting gained PCA explained-variance and component plots with grid embeddings, and var-mask support was added across PCA/NMF/CP. Mismatch-frequency plotting gained base-frequency panels and ambiguous GpC/CpG site handling. This version also added the repo's AGENTS.md/CLAUDE.md agent instruction files and fixed a number of unit/smoke tests.

Version 0.3.1#

0.3.1 2026-01-27 Version 0.3.1 replaced scanpy-based latent embeddings with umap-learn/pynndescent and fixed categorical color mapping in embedding plots, and added CP (CANDECOMP/PARAFAC) tensor factorization with backend selection alongside NMF-based decomposition for the spatial pipeline. Preprocessing gained sequence mismatch annotation, per-reference base-quality statistics, and quality-normalized mismatch z-score plots. A rolling nearest-neighbor distance tool was added for spatial analysis, along with zero-Hamming span detection, clustermaps, and CSV exports (early groundwork for chimera detection). HMM plotting gained per-feature colormaps and length-range overlays, alignment records started tracking secondary/supplementary spans, and logging became reconfigurable per workflow.

Version 0.3.0#

0.3.0 2026-01-20 Version 0.3.0 Has split the smftools dependencies into its core dependencies, as well as various optional dependencies depending on the user's intended use in order to keep the package as light as possible. The package can now use wither a CLI or a Python based backend for Samtools, Bedtools, and BedGraphToBigWig for smftools load functionality.

Version 0.2.5#

0.2.5 2026-01-15 Version 0.2.5 was a major documentation and infrastructure release: Read the Docs was set up and configured, docstrings were added across the analysis utilities and API modules, and CLI usage documentation was expanded. An AnnData history/schema recording structure was established, logging helpers were adopted across subpackages, and CI gained pytest smoke/e2e tests plus format/lint checks.

Version 0.2.4#

0.2.4 2026-01-07 Version 0.2.4 introduced pytest markers and adjusted CI to test against Python 3.10+, along with assorted test fixes.

Version 0.2.3#

0.2.3 2025-12-02 Version 0.2.3 provides the core smftools functionality through several command line commands (load, preprocess, spatial, hmm).

Version 0.2.2#

0.2.2 2025-12-01 Version 0.2.2 added batch and concatenate CLI commands and consolidated CLI entry-point functionality, alongside debugging fixes for the direct-modification-detection workflow.

Version 0.2.1#

0.2.1 2025-11-05 Version 0.2.1 makes the core workflow (smftools load) a command line tool that takes in an experiment_config.csv file for input/output and parameter management.

Version 0.2.0#

0.2.0 2025-08-06 Version 0.2.0 added support for deaminase-based SMF, introduced a CLI for the informatics module, and reorganized the HMM and machine-learning modules.

Version 0.1.7#

0.1.7 2025-07-03 Version 0.1.7 added model-interpretability tooling (SHAP, plus interpretability modules for both sklearn and torch.nn models), GradCAM support for CNNs, and PR/ROC metrics with a more flexible CNN architecture. Sliding-window train/test/inference wrappers for both Lightning and sklearn backends were added, along with broader training, inference, and post-inference evaluation updates.

Version 0.1.6#

0.1.6 2025-05-29 Informatics, preprocessing, tools, plotting modules have core functionality that is approaching stability on MacOS (Intel/Silicon) and Linux (Ubuntu). Documentation/tutorials are still being improved. The base PyTorch/Scikit-Learn ML-infrastructure is going through organizational changes to work with PyTorch Lightning, Hydra, and WanDB to facilitate scaling, multi-device usage, and logging.

Version 0.1.5#

0.1.5 2025-05-29 Version 0.1.5 reorganized the machine-learning modules, adding transformer-based annotation, a scaling wrapper, and basic DANN (domain-adversarial) organization, alongside classifier functionality, plotting, and Bayesian statistics. Preprocessing gained duplicate detection and HMM-based peak calling (including nucleosome HMMs), support for already-demultiplexed input files, and further parallel-processing updates.

Version 0.1.4#

0.1.4 2025-02-24 Version 0.1.4 began adding multiprocessing and GPU utilization to the pipeline, with performance improvements to preprocessing and continued troubleshooting of the conversion SMF module.

Version 0.1.3#

0.1.3 2025-01-28 Version 0.1.3 introduced HMM-based feature annotation, added preprocessing consensus-sequence functionality, and extended demultiplexing support. Conversion SMF patches and informatics/tools optimizations landed alongside new bedgraph/bigwig output support and expanded informatics workflow diagrams.

Version 0.1.2#

0.1.2 2024-09-18 Version 0.1.2 focused on informatics performance and QC: added plotting-based QC for the informatics pipeline, extended informatics functionality, and introduced preprocessing recipes. Added FASTA subsampling by BED interval, along with documentation and README updates.

Version 0.1.1#

0.1.1 2024-09-09 The informatics module has been bumped to alpha-phase status. This module can deal with POD5s and unaligned BAMS from nanopore conversion and direct SMF experiments, as well as FASTQs from Illumina conversion SMF experiments. Primary output from this module is an AnnData object containing all relevant SMF data, which is compatible with all downstream smftools modules. The other modules are still in pre-alpha phase. Preprocessing, Tools, and Plotting modules should be promoted to alpha-phase within the next month or so.

Version 0.1.0#

0.1.0 2024-08-30 smftools initial release. Pre-Alpha phase. J McKenna

Currently, this package (smftools-0.1.0) is going through rapid improvement (dependency handling accross Linux and Mac OS, testing, documentation, debugging) and is still too early in development for widespread use. The underlying functionality was originally developed as a collection of scripts for single molecule footprinting (SMF) experiments in our lab, but is being packaged/developed to facilitate the expansion of SMF to any lab that is interested in performing these styles of experiments/analyses. The alpha-phase package is expected to be available within a couple months, so stay tuned!