Topography UMAP API¶

Project components from a newly-trained model into the published pan-cancer topography UMAP — a shared space built from each component’s normalized mutation-rate profile across chr2 10 kb bins.

Requires the optional umap-learn dependency:

pip install mutopia[umap]

Quick start¶

import mutopia.analysis as mu

# Layout only: ~15 kB, pandas alone, no umap-learn needed.
coords = mu.load_reference_coordinates()

# The full projector.
ref = mu.load_reference_umap()

data = mu.gt.load_dataset("my_cohort.annotated.nc", with_samples=False)
ref.transform(data)        # coordinates, topo-cluster, novelty score
ref.kneighbors(data, k=5)  # nearest reference components

Projecting a model trained on a different grid¶

The reference feature space is defined on a specific ordered locus axis — the same regions bed and the same mesoscale features. A gTensor built differently cannot be matched to it, because mesoscale-split loci share (chrom, start) and no stored column separates them.

A model, however, is a function of genomic features, so evaluate it on a reference-axis gTensor instead and the projection stays on the strict path:

model = mu.load_model("my_cohort.model.pkl")
grid = mu.gt.load_dataset(reference_nc, with_samples=False, with_state=False)

ref.transform_model(model, grid)
ref.kneighbors_model(model, grid, k=5)

# The annotated gTensor itself, e.g. for mutopia.plot.track_plot
annotated = mu.annot_component_rates(model, grid)

Pass with_state=False: a previous model’s corpus state must be shed, and the pan-cancer models convolve each feature over neighbouring bins, leaving a 51-long feature coordinate that a different model will reject.

Interpreting the output¶

The 2D coordinates are for display. transform is an approximate re-optimization of new points against a frozen layout of a few hundred points. The nearest-neighbour table and the cluster vote — both computed in the full-dimensional cosine space — are the defensible readouts.

Two failure modes are reported explicitly, because neither is visible in the coordinates:

is_outlier / outlier_score: A local outlier factor over the reference. Structureless input (noise, shuffled loci) still lands somewhere plausible and receives a confident cluster call; this is what catches it. Nearest-neighbour distance does not, because a bland profile sits near the centroid where the reference is densest.
set_is_degenerate / set_spread: A set-level check. Components that are near-identical to one another each land beside whatever reference point is closest to the centroid, and every per-component statistic looks normal. This fires when the projected set’s median pairwise cosine distance falls below that of the least diverse reference cohort.

Limitations¶

The reference space is chr2 only; a new gTensor must span it.
Cluster labels are hand-curated, so the k-NN vote’s in-sample ceiling is ~0.82, not 1.0.
on_grid_mismatch="aggregate" exists for gTensors that cannot match the axis, but is lossy — pairwise cosine distances correlate r = 0.89 with the raw axis and only 61% of components keep the same nearest neighbour. It warns at runtime. Prefer the model route above.

Reference¶

class mutopia.analysis.topography_umap.TopographyUMAP(n_neighbors=5, min_dist=0.1, random_state=101, negative_sample_rate=3, metric='cosine', n_components=2, cluster_n_neighbors=10, on_grid_mismatch='error', min_bin_coverage=0.9, outlier_quantile=0.99)[source]¶

Bases: BaseEstimator

Reference UMAP of component topographies, with projection of new components.

Parameters:

n_neighbors – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
min_dist – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
random_state – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
negative_sample_rate – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
metric – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
n_components – Passed straight to umap.UMAP. The defaults are the parameters the pan-cancer reference space was built with.
cluster_n_neighbors (int, default=10) – Neighbours used for the k-NN cluster vote. The vote happens in the full feature space, matching how the reference clusters were defined.
on_grid_mismatch ({"error", "aggregate"}, default="error") – What to do when an incoming dataset’s locus axis does not reproduce the reference axis. "error" raises and explains. "aggregate" falls back to a second, lossy reference space in which mesoscale-split loci are pooled into unique (chrom, start) bins – see _aggregate_space() for the measured cost.
min_bin_coverage (float, default=0.9) – Under "aggregate", the minimum fraction of aggregated reference bins that must receive data before the projection is refused.
outlier_quantile (float, default=0.99) – Quantile of the reference’s own local-outlier-factor scores used as the is_outlier threshold.

reducer_¶

The fitted reducer. reducer_.embedding_ is the reference layout.

Type:: umap.UMAP

X_ref_¶

Type:: ndarray of shape (n_reference, n_bins)

embedding_¶

Type:: ndarray of shape (n_reference, n_components)

components_¶

Type:: ndarray of str

metadata_¶

Type:: DataFrame indexed by component

bin_chrom_, bin_start_, bin_length_

The reference locus axis. Note that (chrom, start) is not unique: mesoscale features split a bin into several loci sharing coordinates. The axis is matched positionally – see _axis_matches().

Type:: ndarray

build_features(dataset, source=None)[source]¶

Turn a gTensor into a reference-aligned feature matrix.

Returns:

X (ndarray of shape (n_components, n_bins), float32)
components (ndarray of str)
space ({“strict”, “aggregate”}) – Which reference space X lives in.

features_from_model(model, dataset, source=None, threads=1)[source]¶

Reference-aligned features for a trained model, via annot_component_rates().

Use when the model was trained on a different regions bed than the reference: evaluating it on a reference-axis gTensor keeps the projection on the strict path.

fit(data, metadata=None, source=None, bins=None, embedding=None)[source]¶

Fit the reference space.

Parameters:

data (gTensor Dataset, or (X, components) tuple) – When a tuple is passed, bins must give the (chrom, start, length) locus axis its columns correspond to.
metadata (DataFrame, optional) – Indexed by component; tumor_type/class/cluster_id/ cluster_name columns are carried into the outputs.
embedding (ndarray, optional) – Pin the reference layout to these coordinates instead of the fitted ones. Used to preserve a published layout that a current umap-learn no longer reproduces bit-for-bit; transform optimizes new points against embedding_ held fixed, so any fixed layout of the reference points defines a valid projection.

kneighbors(data, k=10, source=None)[source]¶

Nearest reference components for each input component (long form).

This is the most robust readout – it depends only on cosine distance in the original feature space, not on UMAP’s approximate transform.

kneighbors_model(model, dataset, k=10, source=None, threads=1)[source]¶: Nearest reference components for a trained model’s components.

classmethod load(path, verify=True, atol=0.001)[source]¶: Rebuild a reference space from a .npz written by save().

predict_cluster(data, source=None)[source]¶: k-NN vote for the topography cluster, in the full feature space.

save(path, provenance=None, dtype=<class 'numpy.float16'>)[source]¶

Write the reference space to a compressed .npz.

The fitted umap.UMAP is deliberately not pickled – it carries a pynndescent index with numba-typed internals whose unpickling is fragile across umap/numba versions. load_reference_umap() reconstructs the reducer by refitting on the stored matrix, which is cheap at reference scale and version-robust.

set_fit_request(*, bins='$UNCHANGED$', data='$UNCHANGED$', embedding='$UNCHANGED$', metadata='$UNCHANGED$', source='$UNCHANGED$')¶

Request metadata passed to the fit method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to fit.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters:

bins (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for bins parameter in fit.
data (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for data parameter in fit.
embedding (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for embedding parameter in fit.
metadata (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for metadata parameter in fit.
source (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for source parameter in fit.

Returns:

self – The updated object.

Return type:

object

static set_spread(X)[source]¶: Median pairwise cosine distance within a set of components.

set_transform_request(*, data='$UNCHANGED$', source='$UNCHANGED$')¶

Request metadata passed to the transform method.

Note that this method is only relevant if enable_metadata_routing=True (see sklearn.set_config()). Please see User Guide on how the routing mechanism works.

The options for each parameter are:

True: metadata is requested, and passed to transform if provided. The request is ignored if metadata is not provided.
False: metadata is not requested and the meta-estimator will not pass it to transform.
None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.
str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Note

This method is only relevant if this estimator is used as a sub-estimator of a meta-estimator, e.g. used inside a Pipeline. Otherwise it has no effect.

Parameters:

data (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for data parameter in transform.
source (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for source parameter in transform.

Returns:

self – The updated object.

Return type:

object

transform(data, source=None)[source]¶

Project components into the reference layout.

Returns:

DataFrame indexed by component with the embedding coordinates, the
predicted cluster, the nearest reference component and its cosine
distance, and a local-outlier-factor novelty score with its flag.

transform_model(model, dataset, source=None, threads=1)[source]¶

Project a trained model’s components, evaluating it on dataset’s axis.

>>> ref = mu.load_reference_umap()
>>> model = mu.load_model("my_cohort.model.pkl")
>>> grid = mu.gt.load_dataset(reference_nc, with_samples=False, with_state=False)
>>> ref.transform_model(model, grid)

property umap_params¶

write_coordinates(path)[source]¶: Write the coordinates-only fixture read by load_reference_coordinates().

mutopia.analysis.topography_umap.load_reference_umap(path=None, verify=False, download=True)[source]¶

Load the pan-cancer topography UMAP reference.

The ~11 MB feature matrix is a release asset, not a committed file, so the first call downloads it and caches it under ~/.cache/mutopia (override with MUTOPIA_CACHE_DIR). Subsequent calls are local. A source checkout that has already built the artifact in place uses that copy instead.

If you only need the layout – to draw it, or to check coordinates – use load_reference_coordinates(), which ships in the wheel and needs neither the download nor umap-learn.

Parameters:

path (str, optional) – Explicit artifact to load, bypassing discovery and download.
verify (bool, default=False) – Refit UMAP on the reference matrix and warn if this environment does not reproduce the stored layout. Off by default because it doubles load time.
download (bool, default=True) – Whether to fetch the artifact when it is not already present.

mutopia.analysis.topography_umap.load_reference_coordinates(path=None)[source]¶

The canonical reference layout as a DataFrame – coordinates only.

The full artifact carries an 11 MB feature matrix and needs umap-learn to rebuild its reducer, which is a lot to pay for drawing the backdrop of a plot or asserting against fixed coordinates in a test. This reads a small TSV with pandas alone.

Use it to plot the reference space or to check published coordinates; use load_reference_umap() when you need to project new components.

Returns:

DataFrame indexed by component, with tumor_type, class, cluster_id,
cluster_name, UMAP1 and UMAP2.

mutopia.analysis.topography_umap.annot_component_rates(model, dataset, source=None, threads=1, key='component_distributions_locus')[source]¶

Evaluate a trained model’s component locus rates on any gTensor’s axis.

model.annot_component_distributions routes through setup_corpus, which initializes the locals model and therefore iterates the dataset’s samples. Datasets written with write_samples=False have no raw group, so that path raises ValueError: Sample ... not found — for every model, not just unusual ones. Component locus distributions depend only on the factor model (context + theta), so this initializes just that half and skips the sample-dependent step.

The point of doing this is comparability. A model is a function of genomic features, so evaluating it on the reference gTensor puts its components on the reference locus axis — which lets TopographyUMAP.transform() take the strict path instead of the lossy aggregate one, even when the model was trained on a completely different regions bed.

Parameters:

model (TopographyModel) – A trained model, e.g. from mu.load_model(...).
dataset (gTensor Dataset) – The axis to evaluate on. Load it with with_state=False so the previous model’s corpus state is shed (mu.gt.load_dataset(path, with_samples=False, with_state=False)).
source (str, optional) – Cell type / source for multi-source models.

Returns:

The dataset with component_distributions_locus added, dims
(component, locus) and component coordinates taken from
model.component_names. Suitable for
TopographyUMAP.transform() and for mutopia.plot.track_plot.