Clusterer backends

atom_hifi/training/cluster_backends.py — pluggable clusterer backends (ADR-0009).

The backend is the clusterer only: the PCA embedding is computed once per element (decaf’s get_embed(dim=0)) and shared by every backend via PCAEmbedder. fit_cluster_backend() dispatches to one of the built-in arms and returns a ClusterFit.

Backends

• decaf (distance): mean-shift at bandwidth δ; classifies by native 1-NN radius abstention (OOD/patient/loop support).

• kmeans (count): δ is the cluster count k; pinned n_init=10. The fit calibrates lambda_open — the DP-means cluster-opening price in SQUARED-distance units (gitlab#80) — and ClusterFit.classify abstains via the nearest-centroid rule (OOD/patient/loop support, gitlab#81).

• hdbscan (distance): δ is cluster_selection_epsilon; identical constructor params to decaf’s own get_HDBSCAN (min_samples = min_cluster_size, allow_single_cluster=True), but raw sklearn labels (−1 = noise; decaf’s re-classify step is skipped because its LabelAll branch is broken). Noise atoms are deduped at ε_noise and promoted to first-class groups before fit_cluster_backend() returns.

Issue #69 moved the backend-agnostic pieces to atom_hifi.training.cluster_base and the group ordering to atom_hifi.training.group_order. _PrecomputedDecaf deliberately stays here because its import path is load-bearing: Judge.save pickles whatever a ClusterFit holds, and between issues #68 and #76 that included a live _PrecomputedDecaf. Since #76 no fit retains one, so new pickles do not contain it — but the path must keep resolving or every pickle written in that window becomes unloadable.

class atom_hifi.training.cluster_backends.DecafClassifier(decaf_cluster)

Bases: Classifier

1-NN label transfer with radius abstention, delegated to decaf.

No longer on any production path (gitlab#76): fits now carry an abstain_radius and ClusterFit.classify builds a NearestNeighborClassifier.

Retained deliberately, as the reference the native port is checked against — test_native_classifier_matches_decaf_exactly fits real decaf and asserts the two agree slot-for-slot. Delete this class and that test can no longer distinguish a faithful port from a plausible one. It also keeps any explicitly-supplied ClusterFit.classifier working.

classify(embedded, **kwargs)

Return decaf’s (gID_distinct, gID_label_all) pair.

Both lists have one slot per selection group plus a trailing slot. gID_distinct[-1] holds rejected rows. DECAF and KMeans provide unconditional assignments in gID_label_all (empty trailing slot). HDBSCAN has no unconditional assignment: rejected rows also occupy its second list’s trailing slot. Promoted noise slots stay empty.

class atom_hifi.training.cluster_backends.DecafBackend

Bases: ClusterBackend

Mean-shift at bandwidth δ with 1-NN radius classification.

name: str = 'decaf'

cache tag — must be stable across runs (state_io enforces equality)

granularity_kind: str = 'distance'

‘distance’ (δ is a radius) | ‘count’ (δ is a cluster count k)

has_classifier: bool = True

fit yields a deployable classifier → OOD / patient / loop

has_auto_bandwidth: bool = True

fit yields its own bandwidth estimate (decaf’s HaarWavelet δ_auto)

fit(desc_el, granularity, *, t_embedded=None, **kw)

Cluster one element; return an unpromoted ClusterFit.

Takes raw desc_el rather than a ready embedding because decaf builds its own embedding internally — inject_precomputed needs the decaf object itself. Backends that cluster in the shared embedding should call shared_embedder() and use its t_embedded; that is the one-liner a custom backend needs.

class atom_hifi.training.cluster_backends.KMeansBackend

Bases: ClusterBackend

k-means in the shared PCA embedding; δ is the cluster count k.

name: str = 'kmeans'

cache tag — must be stable across runs (state_io enforces equality)

granularity_kind: str = 'count'

‘distance’ (δ is a radius) | ‘count’ (δ is a cluster count k)

has_classifier: bool = True

fit yields a deployable classifier → OOD / patient / loop

fit(desc_el, granularity, *, t_embedded=None, seed=42, **kw)

Cluster one element; return an unpromoted ClusterFit.

Takes raw desc_el rather than a ready embedding because decaf builds its own embedding internally — inject_precomputed needs the decaf object itself. Backends that cluster in the shared embedding should call shared_embedder() and use its t_embedded; that is the one-liner a custom backend needs.

class atom_hifi.training.cluster_backends.HDBSCANBackend

Bases: ClusterBackend

Sklearn density clustering with frozen hierarchy deployment (issue #97).

δ remains cluster_selection_epsilon for fitting. Prediction uses the fitted density tree; promoted noise groups never become deployment targets.

name: str = 'hdbscan'

cache tag — must be stable across runs (state_io enforces equality)

granularity_kind: str = 'distance'

‘distance’ (δ is a radius) | ‘count’ (δ is a cluster count k)

emits_noise: bool = True

fit can leave a trailing noise slot needing ε_noise promotion (#63)

has_classifier: bool = True

fit yields a deployable classifier → OOD / patient / loop

classmethod unsupported_reason(granularity=None)

Why this backend cannot run in this environment, or None if it can.

The capabilities above are static facts about the algorithm; this one is not. It reports a dependency fault — a broken upstream version, a missing optional package — so it has to be asked at runtime rather than read off the class.

Two questions, one method:

• granularity=None asks the general one — can this backend be relied on here at all? Scan entry points and the test skip marker use this, because a scan sweeps δ and will reach the bad values.

• passing a δ asks about that one fit, which lets a backend admit the δ values it can still serve. fit_cluster_backend uses this, so a dependency fault that only bites part of the δ range does not take the whole backend down with it.

fit(desc_el, granularity, *, t_embedded=None, min_cluster_size=2, **kw)

Cluster one element; return an unpromoted ClusterFit.

Takes raw desc_el rather than a ready embedding because decaf builds its own embedding internally — inject_precomputed needs the decaf object itself. Backends that cluster in the shared embedding should call shared_embedder() and use its t_embedded; that is the one-liner a custom backend needs.

atom_hifi.training.cluster_backends.BACKEND_REGISTRY = {'decaf': <class 'atom_hifi.training.cluster_backends.DecafBackend'>, 'hdbscan': <class 'atom_hifi.training.cluster_backends.HDBSCANBackend'>, 'kmeans': <class 'atom_hifi.training.cluster_backends.KMeansBackend'>}

name → backend class. The single source of truth for what backends exist and what they can do; GRANULARITY_KIND and BUILTIN_BACKENDS are derived from it so they cannot drift.

atom_hifi.training.cluster_backends.register_backend(backend)

Register a custom ClusterBackend under its name.

Registration makes the backend resolvable by name in this process, which is what config validation and state_io cache tagging need. It does NOT reach process workers — the child interpreter re-imports the module and gets only the built-ins — so the executable spec is additionally threaded through the worker kwargs (HiFiWorkflow._backend_spec).

The backend is stored as given. In particular, an instance is not replaced by its class: constructor state is retained and required arguments need not be reconstructed. That registered instance is a configured prototype; resolution deep-copies it so every fit starts from the same state.

Returns the registered name, so callers can store the tag.

atom_hifi.training.cluster_backends.resolve_backend(backend)

Resolve a name, subclass, or configured prototype to a fresh instance.

atom_hifi.training.cluster_backends.fit_cluster_backend(backend, desc_el, delta, *, t_embedded=None, min_cluster_size=2, seed=42, eps_noise=None)

Fit one clustering backend for one element; return a ClusterFit.

The returned fit is already promoted: any trailing noise/distinct slot has been deduped at eps_noise and appended as real groups (issue #63), so every consumer sees one uniform group namespace with no -1 atoms.

Parameters:

• backend ({‘decaf’, ‘kmeans’, ‘hdbscan’} or a ClusterBackend)

• desc_el ((N_el, D) ndarray — raw per-element descriptors.)

• delta (float — granularity: bandwidth δ (decaf), cluster count k (kmeans),) – cluster_selection_epsilon (hdbscan).

• t_embedded (ndarray, optional — precomputed PCA embedding (SHM fast path).)

• min_cluster_size (int — hdbscan only.)

• seed (int — kmeans only (n_init=10 pinned).)

• eps_noise (float, optional — dedup radius for trailing-slot promotion.) – Required only when the backend actually produces a trailing slot (hdbscan); kmeans and decaf leave it empty, so promotion is a no-op.

atom_hifi.training.cluster_backends.GRANULARITY_KIND = {'decaf': 'distance', 'hdbscan': 'distance', 'kmeans': 'count'}

backend name → ‘distance’ | ‘count’. Derived; do not hand-maintain. register_backend() extends both this and BACKEND_REGISTRY together.

atom_hifi.training.cluster_backends.BUILTIN_BACKENDS = ('decaf', 'kmeans', 'hdbscan')

the three shipped backends — frozen, and NOT extended by register_backend. Validation checks BACKEND_REGISTRY (which grows); this tuple stays the answer to “what does atom-hifi ship”, e.g. for name-collision checks.