Frozen HDBSCAN deployment

Design established before implementation for issue #97.

What an accepted environment means

An accepted query attaches to a density cluster in the frozen reference hierarchy, in the same per-element PCA coordinates used for selection. It does not create a cluster, change reference density, or establish a force-error bound. OOD means that this attachment is unsupported by the selected density partition. It is conditional on the reference library and its granularity.

The fitting engine and parameters remain sklearn HDBSCAN, Euclidean distance, min_samples = min_cluster_size, EOM selection, allow_single_cluster=True, and the existing cluster_selection_epsilon scan. Labels, medoids, noise promotion, representative selection, and L/R/F retain their existing meaning. No switch to the contrib fitting engine is required.

Density attachment rule

  1. Freeze each reference point’s core distance: distance to its mth nearest reference point, including itself, where m = min_samples in sklearn. Reconstruct these distances with the same KD-tree path as sklearn’s dense Euclidean fit, avoiding a numerically different brute-force calculation.

  2. For a novel query, its core distance is the distance to its (m-1)th nearest reference point: the query supplies its own zero-distance self. Queries never count each other, so batching cannot change density.

  3. Find the reference point minimizing mutual reachability max(query_core, reference_core, Euclidean_distance). Search the whole reference, with bounded distance blocks. Resolve equal minima by reference row order. Convert this attachment distance to lambda (inverse distance; zero distance gives positive infinity).

  4. Cap attachment lambda at the anchor’s departure lambda in the condensed tree. Climb to the ancestor present at that density, then resolve its selected ancestor using the actual sklearn fitted labels, including epsilon merges. Reject an attachment outside a selected cluster. At a non-root cluster’s birth boundary, use the parent (strict birth test).

  5. A selected root needs an additional support boundary because it has no birth split. Use sklearn’s own root membership rule: attachment lambda must reach the root’s maximum outgoing lambda when epsilon is zero, or 1 / cluster_selection_epsilon otherwise. The latter is specifically sklearn’s selected-root density cut, not a general OOD radius.

This is Atom-HiFi’s frozen hierarchy attachment policy, not a claim of equivalence to refitting HDBSCAN with queries appended or to contrib approximate_predict. Reference core distances and hierarchy stay frozen. In particular, a query equal to a reference row is still evaluated as a query; training-label reproduction at density ties is not guaranteed.

Selection groups versus deployment groups

ClusterFit.n_groups continues to count all coverable groups. A promoted noise representative keeps its group and noise_derived=True for selection and inspection. It does not become a supported deployment target. n_deployable_groups counts the selected density groups with prediction state; an all-noise fit is a valid predictor with zero deployment targets and rejects every query.

ClusterFit.classify returns two lists with n_groups + 1 slots. For HDBSCAN, both lists contain accepted indices in backend-local density slots and rejected indices in the trailing slot; promoted slots are empty. There is no forced label for rejection. KMeans and DECAF retain their existing second-list unconditional assignment convention. Consumers needing OOD always use the first list’s trailing slot. Saved-Judge CSV output maps accepted labels through the saved permutation and element offset; rejected HDBSCAN rows have group_id=-1, ood=1.

Persistence and compatibility

Persist versioned numeric prediction state, the PCA projection, reference core distances, condensed-tree ancestry/departure scales, and selected-node-to-local label mapping. No fitted sklearn estimator or neighbor index is needed at deployment. Loading and predicting must not fit HDBSCAN or PCA. The sklearn private-tree adapter is confined to fitting and must validate its reconstructed partition against sklearn’s labels before exposing a predictor.

A legacy HDBSCAN fit without this state remains usable for inspection and selection. Classification raises an actionable error requesting an explicit new fit from the original reference descriptors; loading never repairs it by refitting. Selection caches are not deployable Judges.

On the scan path the existing PCA projection may be lazy. Judge.save materializes it explicitly before writing an HDBSCAN Judge. Generic pickle serialization and joblib transport do not compute it. This keeps repeated scan fits cheap while ensuring the saved-Judge prediction path needs only a matrix multiplication. The numeric hierarchy state has its own schema version; unknown schemas fail classification rather than silently changing semantics.

Prediction uses exact distances to the reference, so work scales as N_query × N_reference × embedding_dimension. Distance blocks bound working memory; queries can also be streamed in external batches with identical results. The hierarchy and core distances are stored once. At float64, the reference array shares the embedder’s storage and pickle memoization avoids a second copy.

Run the saved-Judge example

Use the feature checkout explicitly; do not reinstall the package in the shared environment. The output directory must be new.

cd /work1/SKTable_Injector/atom-hifi-hdbscan-deploy
export PYTHONPATH="$PWD"
export PYTHONDONTWRITEBYTECODE=1
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1
/home/yhsong/.virtualenvs/test/bin/python -u scripts/example_hdbscan_deployment.py \
  --epsilon 0.05 --output-dir /tmp/hdbscan-saved-judge-example

# A separate process loads the saved fit and classifies its reference descriptors.
/home/yhsong/.virtualenvs/test/bin/python -u scripts/predict_ood.py \
  --judge /tmp/hdbscan-saved-judge-example/judge_final.pkl \
  --desc /tmp/hdbscan-saved-judge-example/reference_desc.pkl \
  --lib /tmp/hdbscan-saved-judge-example/reference.xyz \
  --out /tmp/hdbscan-saved-judge-example/prediction-reloaded.csv

The example uses 20 checked-in Ni–O structures (317 atoms), generates ED-SOAP, fits a fixed HDBSCAN operating point and the N//2 microstate reference, saves the refined Judge/mapping, and compares CSV versus patient OOD after loading. Its summary.json includes selection and deployment counts separately. These are reference-only checks, not a held-out accuracy or lifetime study.

Integrating the later HDBSCAN lifetime arm

  1. Use branch feat/native-hdbscan-deployment in a separate checkout. The implementation baseline is 2d5544ec6231bdbe1bc3996fc8f6f9c099346f45. Keep the active KMeans checkout and shared environment unchanged:

    git -C /work1/SKTable_Injector/atom-hifi fetch origin feat/native-hdbscan-deployment
git -C /work1/SKTable_Injector/atom-hifi worktree add \
  -b study/hdbscan-lifetime /work1/SKTable_Injector/atom-hifi-hdbscan-lifetime \
  origin/feat/native-hdbscan-deployment
cd /work1/SKTable_Injector/atom-hifi-hdbscan-lifetime
export PYTHONPATH="$PWD"
export PYTHONDONTWRITEBYTECODE=1
/home/yhsong/.virtualenvs/test/bin/python -c \
  'import atom_hifi; print(atom_hifi.__file__)'

    The printed module path must be this HDBSCAN checkout. Record its exact git rev-parse HEAD in the arm’s manifest. The validated environment is Python 3.12, sklearn 1.8.0 and NumPy 2.2.2; no dependency change is required. The existing NumPy >=2.4 positive-epsilon fitting guard remains in force.

  2. For each lifetime reference interval, fit a new reference-only HDBSCAN Judge using the arm’s established descriptor configuration, focus atoms, PCA path, microstate reference, scan and operating-point rule. Set cluster.backend: hdbscan, the intended cluster.min_cluster_size, and output.save_judge: true; give the arm a fresh paths.output_dir. HDBSCAN keeps the c_factor/epsilon axis; a KMeans k or lambda_open is not an HDBSCAN operating point. Keep reference/query membership and ordering matched to the study protocol. Query descriptors must not enter fitting or operating-point selection. Do not reuse KMeans clustering/selection caches or promote its saved Judge by changing a backend name. Shared descriptor inputs may be read only when their configuration and row mapping match.

  3. Save after judge.apply_refined_group_ids() when constructing Judges directly; normal plot_final_fit already does this. Save the associated mapping CSV. A legacy HDBSCAN Judge or el_fit_cache.npz lacks deployment state and requires an explicit reference refit in the new output directory.

  4. Load that interval’s Judge once and freeze it for every later query batch:

    import numpy as np
from atom_hifi import Judge

judge = Judge.load("HDBSCAN_ARM/reference_interval/judge_final.pkl")
fit = judge.cluster_per_el[element]
slots, _ = fit.classify(fit.embedder.embed(query_descriptors))
labels = np.full(len(query_descriptors), -1, dtype=int)
for backend_id, rows in enumerate(slots[:-1]):
    labels[rows] = backend_id
ood_count = len(slots[-1])
accepted = labels >= 0
global_ids = np.full(len(labels), -1, dtype=int)
order = judge.group_order_per_el[element].local_of_old
global_ids[accepted] = (
    order[labels[accepted]] + judge.group_id_offset[element])

    Alternatively use scripts/predict_ood.py --judge ... --desc ... --lib ... on each batch; it applies the same mapping. Mode B (--config ... --xyz ...) computes descriptors with the reference run’s descriptor settings.

  5. Report atom OOD over all queried focus atoms. Keep fit.n_groups, fit.n_density_groups, promoted count, and fit.n_deployable_groups separate. If reporting visited-state coverage, use deployable density groups as the denominator; an all-noise reference has zero such states, so state coverage is undefined, not 100%. Its atom OOD is 100% for a nonempty query. Promoted groups retain their selection costs and L/R/F contribution. Group IDs are local to one fitted reference and cannot be matched across intervals as chemical identities.

  6. Run this bounded example and the focused deployment tests before launching the arm under the study’s own resource and stopping rules. No PIT-WATER lifetime run is part of this implementation. Integration into main still requires explicit approval.

Sources and scope

This feature provides deployment semantics. It does not validate HDBSCAN’s scientific usefulness for PIT-WATER or launch its lifetime arm.