HDBSCAN deployment validation

Issue #97, branch feat/native-hdbscan-deployment, based on 2d5544ec6231bdbe1bc3996fc8f6f9c099346f45. The deployment guide records the semantics established before coding, the runnable example and later-arm integration instructions. Validated with Python 3.12.6, NumPy 2.2.2, SciPy 1.17.0 and sklearn 1.8.0.

Behavioral checks

Final results: 147 focused tests passed (13 slow tests deselected), plus 21 explicit selection and group-order regressions passed. Eleven focused cases are the new density-deployment contracts. Upstream NumPy/sklearn deprecation and duplicate-point KMeans convergence warnings did not fail validation.

The focused suite covers frozen mutual-reachability assignment, epsilon-merged namespaces, accepted and rejected root queries, promoted noise exclusion, all-noise/empty fits, query batching, persistence, legacy missing state, actual refined/global mapping, patient OOD, and the saved-Judge CSV consumer. Persistence tests disable HDBSCAN and PCA fitting after saving. Existing DECAF/KMeans tests retain their previous assignments and mapping schemas.

Review found and resolved a numerical mismatch in core-distance construction: NearestNeighbors(auto) could choose brute force while HDBSCAN used a KD-tree. On a six-row, 16-dimensional fixture shifted by 1e8, cancellation reduced the brute-force core distances to zero and changed the predicted group. The adapter now uses the fitting engine’s KD-tree path; a translation-invariance regression checks the actual prediction rather than the constructor arguments. Independent review reports no unresolved correctness or compatibility findings.

Run from the feature checkout with the specified shared interpreter and no installation step:

export PYTHONPATH="$PWD"
export PYTHONDONTWRITEBYTECODE=1
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1
/home/yhsong/.virtualenvs/test/bin/python -m pytest -q \
  tests/test_hdbscan_deployment.py tests/test_cluster_backends.py \
  tests/test_config.py tests/test_kmeans_abstention.py \
  tests/test_predict_ood_script.py tests/test_judge.py

# Explicitly include these small tests marked slow in the normal suite.
/home/yhsong/.virtualenvs/test/bin/python -m pytest -q -o addopts='' \
  tests/test_cluster_backends.py::test_golden_decaf_regression \
  tests/test_cluster_backends.py::test_judge_kmeans_end_to_end \
  tests/test_cluster_backends.py::test_workflow_run_final_hdbscan \
  tests/test_reps_group_id.py tests/test_group_order.py

The independent review additionally exercised 5,000 small randomized fits across clustered, uniform, curved and duplicate-point data, several min_samples values and epsilon/root regimes. It confirmed selected-partition reconstruction and batch invariance. This is implementation evidence, not held-out scientific validation or a throughput benchmark.

Reference-only canary

The runnable example used the first 20 structures of the checked-in Ni–O fixture: 143 Ni and 174 O atoms. It computed ED-SOAP with existing defaults and the N//2 microstate reference, then fitted HDBSCAN with min_cluster_size=2, seed 42 for selection, and noise-dedup radii Ni 0.012 / O 0.030.

Epsilon

Selected structures

L

R

F

Density/deployable groups Ni / O

Promoted groups Ni / O

OOD atoms Ni / O

0

20

1

1

1

51 / 72

11 / 4

11 / 4

0.05

11

0.8667350258627058

0.9845386922148225

0.8803463314508188

12 / 16

4 / 3

4 / 3

The epsilon 0.05 selection indices are [1, 4, 5, 7, 8, 10, 11, 12, 14, 15, 18].

An archive of the inspected baseline ran separately under /tmp, using the same reference descriptor block and parameters. For both epsilon values, the complete JSON snapshots of fitted labels, centers, noise provenance, group counts, selected indices and L/R/F were byte-identical. Snapshot SHA-256: 406ae00141868f6ed2dc34c5153415fab72cdcda76b1ba26b2079588400d3ff0.

Saved-Judge CSV and patient OOD agreed on every canary. A separate scripts/predict_ood.py process reproduced the epsilon 0.05 CSV byte-for-byte (317 rows, seven rejections). Promoted reference environments can remain OOD: being selected for coverage does not turn them into density-supported targets.

Isolation and limits

  • Worktree: /work1/SKTable_Injector/atom-hifi-hdbscan-deploy.

  • Canary and baseline artifacts: /tmp/atom-hifi-hdbscan-validation/.

  • No changes to the shared Python environment, original checkout, saved study Judges, reference inputs or study caches. No PIT-WATER lifetime arm launched.

  • The original checkout started with the existing untracked report.log.

  • Prediction requires exact query-to-reference distances in bounded blocks; large-reference deployment throughput has not been benchmarked here.

  • The fit-time sklearn private-tree adapter validates its reconstructed labels against sklearn before producing deployment state. The existing NumPy >=2.4 positive-epsilon guard remains; zero-epsilon tests stay enabled there.

  • Sphinx rendered the guide and example. Full documentation builds retain existing _static, unlisted-concepts and state_io docstring diagnostics; external intersphinx inventories were unavailable in the restricted network. No new deployment-page or README-link warning remains.