# HDBSCAN deployment validation

Issue [#97](https://gitlab.mpcdf.mpg.de/yhsong/atom-hifi/-/issues/97), branch
`feat/native-hdbscan-deployment`, based on
`2d5544ec6231bdbe1bc3996fc8f6f9c099346f45`.
The [deployment guide](hdbscan_deployment.md) records the semantics established
before coding, the runnable example and later-arm integration instructions.
Validated with Python 3.12.6, NumPy 2.2.2, SciPy 1.17.0 and sklearn 1.8.0.

## Behavioral checks

**Final results: 147 focused tests passed (13 slow tests deselected), plus 21
explicit selection and group-order regressions passed.** Eleven focused cases
are the new density-deployment contracts. Upstream NumPy/sklearn deprecation
and duplicate-point KMeans convergence warnings did not fail validation.

The focused suite covers frozen mutual-reachability assignment, epsilon-merged
namespaces, accepted and rejected root queries, promoted noise exclusion,
all-noise/empty fits, query batching, persistence, legacy missing state, actual
refined/global mapping, patient OOD, and the saved-Judge CSV consumer. Persistence
tests disable HDBSCAN and PCA fitting after saving. Existing DECAF/KMeans tests
retain their previous assignments and mapping schemas.

Review found and resolved a numerical mismatch in core-distance construction:
`NearestNeighbors(auto)` could choose brute force while HDBSCAN used a KD-tree.
On a six-row, 16-dimensional fixture shifted by `1e8`, cancellation reduced
the brute-force core distances to zero and changed the predicted group.
The adapter now uses the fitting engine's KD-tree path; a translation-invariance
regression checks the actual prediction rather than the constructor arguments.
Independent review reports no unresolved correctness or compatibility findings.

Run from the feature checkout with the specified shared interpreter and no
installation step:

```bash
export PYTHONPATH="$PWD"
export PYTHONDONTWRITEBYTECODE=1
export OMP_NUM_THREADS=1 OPENBLAS_NUM_THREADS=1 MKL_NUM_THREADS=1
/home/yhsong/.virtualenvs/test/bin/python -m pytest -q \
  tests/test_hdbscan_deployment.py tests/test_cluster_backends.py \
  tests/test_config.py tests/test_kmeans_abstention.py \
  tests/test_predict_ood_script.py tests/test_judge.py

# Explicitly include these small tests marked slow in the normal suite.
/home/yhsong/.virtualenvs/test/bin/python -m pytest -q -o addopts='' \
  tests/test_cluster_backends.py::test_golden_decaf_regression \
  tests/test_cluster_backends.py::test_judge_kmeans_end_to_end \
  tests/test_cluster_backends.py::test_workflow_run_final_hdbscan \
  tests/test_reps_group_id.py tests/test_group_order.py
```

The independent review additionally exercised 5,000 small randomized fits
across clustered, uniform, curved and duplicate-point data, several
`min_samples` values and epsilon/root regimes. It confirmed selected-partition
reconstruction and batch invariance. This is implementation evidence, not
held-out scientific validation or a throughput benchmark.

## Reference-only canary

The runnable example used the first 20 structures of the checked-in Ni–O
fixture: 143 Ni and 174 O atoms. It computed ED-SOAP with existing defaults and
the N//2 microstate reference, then fitted HDBSCAN with `min_cluster_size=2`,
seed 42 for selection, and noise-dedup radii Ni 0.012 / O 0.030.

| Epsilon | Selected structures | L | R | F | Density/deployable groups Ni / O | Promoted groups Ni / O | OOD atoms Ni / O |
|---|---:|---:|---:|---:|---|---|---|
| 0 | 20 | 1 | 1 | 1 | 51 / 72 | 11 / 4 | 11 / 4 |
| 0.05 | 11 | 0.8667350258627058 | 0.9845386922148225 | 0.8803463314508188 | 12 / 16 | 4 / 3 | 4 / 3 |

The epsilon 0.05 selection indices are
`[1, 4, 5, 7, 8, 10, 11, 12, 14, 15, 18]`.

An archive of the inspected baseline ran separately under `/tmp`, using the
same reference descriptor block and parameters. For both epsilon values, the
complete JSON snapshots of fitted labels, centers, noise provenance, group
counts, selected indices and L/R/F were byte-identical. Snapshot SHA-256:
`406ae00141868f6ed2dc34c5153415fab72cdcda76b1ba26b2079588400d3ff0`.

Saved-Judge CSV and patient OOD agreed on every canary. A separate
`scripts/predict_ood.py` process reproduced the epsilon 0.05 CSV byte-for-byte
(317 rows, seven rejections). Promoted reference environments can remain OOD:
being selected for coverage does not turn them into density-supported targets.

## Isolation and limits

- Worktree: `/work1/SKTable_Injector/atom-hifi-hdbscan-deploy`.
- Canary and baseline artifacts: `/tmp/atom-hifi-hdbscan-validation/`.
- No changes to the shared Python environment, original checkout, saved study
  Judges, reference inputs or study caches. No PIT-WATER lifetime arm launched.
- The original checkout started with the existing untracked `report.log`.
- Prediction requires exact query-to-reference distances in bounded blocks;
  large-reference deployment throughput has not been benchmarked here.
- The fit-time sklearn private-tree adapter validates its reconstructed labels
  against sklearn before producing deployment state. The existing NumPy >=2.4
  positive-epsilon guard remains; zero-epsilon tests stay enabled there.
- Sphinx rendered the guide and example. Full documentation builds retain
  existing `_static`, unlisted-concepts and `state_io` docstring diagnostics;
  external intersphinx inventories were unavailable in the restricted network.
  No new deployment-page or README-link warning remains.