# Configuration

Atom-HiFi is config-driven (ADR-0005, local-only). A single commented YAML
template in {py:mod}`atom_hifi.config` is the source of truth for the public CLI
surface: it is written verbatim by `atom-hifi init`, parsed for YAML defaults,
and used as the strict allow-list for validation (unknown keys are rejected).
Expert-only controls for ND grid scan and patient/OOD are accepted through the
Python API, not YAML.

## Build a config for your system

The first job of `config.yaml` is to translate your scientific question into an
Atom-HiFi run: which structures define the library, which atoms define coverage,
which descriptor measures local environments, and which clustering/search
strategy should choose the representative set. Start from the generated
template, fill in the system identity fields, then choose the backend and
multi-element search path deliberately.

Generate a starter file:

```bash
atom-hifi init config.yaml
```

Then edit the minimum system block:

```yaml
paths:
  lib_path: train_structs.xyz   # ASE-readable structure library
  focus_elements: [Ni, O]       # elements whose local environments matter
  output_dir: fr_results_NiO
descriptor:
  kind: eds                     # eds for built-in SOAP, ace for MACE ACE
```

Choose these four values from your system:

| Question | Config key | First-run choice |
|---|---|---|
| Where is the structure library? | `paths.lib_path` | Path to an ASE-readable file such as `.xyz`, `.extxyz`, or trajectory input |
| Which atoms should define coverage? | `paths.focus_elements` | The species whose local environments you want represented; use `null` only when all species matter equally |
| Where should this run write caches and results? | `paths.output_dir` | A fresh directory name for this system and descriptor choice |
| Which descriptor should be used? | `descriptor.kind` | `eds` unless you already have a MACE model and want ACE descriptors |

### Applied example: multi-element adsorbate library

For a NOx/H adsorbate library on a surface, the scientifically relevant local
states may be the adsorbate atoms rather than every support atom. A production
config can therefore focus on N/O/H, use ED-SOAP, let HDBSCAN keep distinct
noise environments, and use NSGA-II to pick separate c-factors for each element:

```yaml
paths:
  lib_path: all_structures_cycle-3.xyz
  focus_elements: [N, O, H]
  output_dir: hdbscan_results
descriptor:
  kind: eds
  eds:
    species: null
    lengthscale: 1.1
    periodic: true
cluster:
  backend: hdbscan
  min_cluster_size: 2
selection:
  method: mu_tiebreak
microstate:
  reference: r100
eps_noise:
  method: thermal
scan:
  l_tol: 0.95
  n_coarse: 60
  n_fine: 60
nsga2:
  run: true
  n_gen: 50
  pop_size: 60
piecing:
  enabled: false
output:
  extract_surroundings: true
```

Read this as a set of modelling choices:

| Config group | What it means in this example |
|---|---|
| `focus_elements: [N, O, H]` | Coverage is judged on adsorbate atoms. Surface/support atoms still appear in descriptors and output structures, but they do not define the set-cover targets. |
| `descriptor.kind: eds` | Use the built-in ED-SOAP descriptor. `lengthscale: 1.1` Å sets the physical scale for the short/long SOAP cutoffs, so it should be near a typical local bond length in the chemistry being resolved. |
| `cluster.backend: hdbscan` | Partition the ED-SOAP embedding with a distance-kind backend that can mark sparse/outlier environments as noise; Atom-HiFi deduplicates those noise environments and forces representatives for them. |
| `microstate.reference: r100` | Use an ε_noise-radius KMeans reference (or `r<NUM>` for a user-defined percentile). This is an opt-in physical assumption: environments inside ε_noise are treated as indistinguishable for the reference. Use the default `n2` when that assumption is uncertain. |
| `nsga2.run: true` | For three focus elements, choose independent c-factors for N/O/H instead of forcing one common resolution scale. |
| `output.extract_surroundings: true` | Write one representative environment per final group, useful for inspecting chemical motifs. |

The remaining groups control runtime, restart behaviour, and inspection output:

| Config group | Applied setting | Meaning |
|---|---|---|
| `n_jobs`, `compute.n_jobs_desc`, `compute.n_jobs_eps` | `n_jobs: 60`, `n_jobs_desc: 60`, `n_jobs_eps: null` | Use 60 workers for general parallel stages and descriptor computation; `n_jobs_eps: null` means ε_noise calibration also uses top-level `n_jobs`. Reduce only after checking memory pressure. |
| `log_level`, `seed` | `info`, `42` | `info` prints stage summaries and progress without debug noise. The seed fixes k-means, scan subsampling, and stochastic optimisation choices. |
| `scan` | `l_tol: 0.95`, `c_factor_range: null`, `n_coarse: 60`, `n_fine: 60` | Keep the default Likeness floor: selected points must preserve at least 95% of full-library microstate entropy. Set `l_tol: null` to disable that floor and auto-pick the maximum-F point. `c_factor_range: null` scans the standard `[0.1, 5.0]` range; coarse/fine counts set scan resolution. |
| `loop` | `run: false` | Skip active-learning and run on the full library. All built-in backends support the loop. OOD need not decay to zero, so a loop may consume the full library. HDBSCAN's promoted noise representatives remain outside its deployable density targets. |
| `nsga2.restore` | `false` | Recompute the Pareto front. Set `true` only to reuse an existing compatible `pareto_front.csv` in the same output directory. |
| `piecing` | `enabled: false` | Treat each input structure as one selectable item. This is right for a normal many-structure adsorbate library. |
| `output.delta_pick` | `null` | Let Atom-HiFi pick the operating point automatically from the scan/NSGA-II result. Set manually only for deliberate refits. |
| `output.save_judge` | `true` | Write the fitted `Judge` to `judge_final.pkl` after the final fit, so the selection can be re-examined without a refit. The pickle omits the raw descriptor block, which only `extract_rep_surroundings` needs; rebuild it from `desc_lib.pkl` — a list of per-structure arrays, so stack it and keep the focus-element rows — and pass it to `judge.attach_descriptors`. Set `false` to skip the file entirely. |
| `surroundings` | `output_mode: full_structure`, `r_cut: null`, `n_per_group: 1` | Save the original periodic structure with center/neighbour/rest tags for one representative per group. `r_cut: null` uses the descriptor cutoff. |
| `refit.delta` | `null` | Run the full scan. Set to a c-factor from `fine_scan.out` only when re-fitting a saved run at a chosen point. |

### Choosing the backend

`cluster.backend` decides how local atomic states are formed from the embedded
descriptor cloud. Pick it from the workflow you need, not from speed alone:

| Backend | Use when | Avoid when |
|---|---|---|
| `decaf` | You want the default, deployable mean-shift path; you may need active-learning loop or patient/OOD classification. | Very large scans where speed is the main bottleneck and you do not need deployable classification. |
| `hdbscan` | You want density clustering with noise representatives for selection and frozen density attachment for deployment. See [prediction semantics](hdbscan_deployment.md). | Noise representatives must be accepted deployment targets: promotion supplies no density support. NumPy 2.4+ remains refused for the positive-epsilon fitting path. |
| `kmeans` | You want a fast count-kind selector, compact covers, NSGA-II over per-element k-fractions, or nearest-centroid OOD/patient/loop classification. | You need grid c-factor optimisation or ε-anchored distance-bandwidth interpretation. Its loop may consume the full library when resolution-tracking OOD remains positive. |

Distance-kind backends (`decaf`, `hdbscan`) sweep `c_factor`, meaning the actual
per-element clustering bandwidth is `δ_el = c_el × ε_noise_el`. Count-kind
backends (`kmeans`) sweep how many clusters to use, so c-factor logic does not
apply; NSGA-II instead optimises an independent per-element k-fraction.

### Multi-element search: why NSGA-II is the default

With more than one focus element, one shared granularity can be too coarse for
one element and too fine for another. `nsga2.run: true` searches per-element
c-factors for distance-kind backends or per-element k-fractions for kmeans, then
tags two useful operating points in `pareto_front.csv`:

| Output tag | Meaning |
|---|---|
| `selected` | Smallest redundancy among points with `L >= scan.l_tol`; this drives the final fit. With `l_tol: null`, it is the `fidelity_best` point. |
| `fidelity_best` | Highest `F = L/R` on the Pareto front; inspect as a candidate, not automatically the conservative choice. |

Use NSGA-II for multi-element production runs with `decaf`, `hdbscan`, or `kmeans`.
Set `pop_size` to at least `10 × N_focus_elements` and at least 40; for
N/O/H, `pop_size: 60` is a reasonable budget. Increase `n_gen` if
`pareto_convergence.png` has not plateaued. Use `nsga2.restore: true` only when
you want to reuse an existing `pareto_front.csv` from the same output directory
and compatible config.

### Microstate reference and ε_noise

For a first run on an unfamiliar system, `microstate.reference: n2` is the
conservative default: per element, it uses roughly two atoms per microstate.
Switch only when the physical assumption is clear:

| Reference | Meaning | Use when |
|---|---|---|
| `n2` | Conservative k=N//2 microstate reference. | Default for unknown, soft, mobile, or mixed systems. |
| `r100` | Smallest resolved k whose KMeans radius is within ε_noise, searched through `N_atoms-1`. Results finer than n2 are kept with an advisory warning. | Known-rigid systems, or analyses where ε_noise is a valid identity radius. |
| `r<NUM>` | Same criterion at user percentile NUM (1–100); `r100` = the maximum. With an explicit `eps_noise.force` block (or `--f-tol`) it resolves against ε_F like `f_tol`, overriding that mode's p95 default; force-resolution then requires thermal ε(σ) as well. | Tuning strictness deliberately. **Hazard:** the historical r90 criterion under-selected k ~3× (+56 % force error on PdMgO) — a thermal `r<NUM>` < 100 is an explicit user opt-in, not a default. |
| `f_tol` | Force-resolution reference from `eps_noise.force.f_tol`, searched up to `N_atoms-1`; requires thermal ε(σ). | Expert MLIP-training studies where force tolerance is the reference scale. |

`eps_noise.method: thermal` is the default and measures descriptor motion under
QHO thermal rattles. Use `precomputed` only when you are carrying known ε values
from a compatible prior run. Use `rdf` when you deliberately want the library's
own first-neighbour peak width to set the displacement scale.

### When to use piecing

`piecing.enabled: true` is for a specific failure mode: a library made of a few
very large structures, often around 1000 atoms each, where every structure
already contains almost every microstate. In that case set-cover can return
`|S| = 1`, the c-factor inflates, and fidelity metrics saturate. Piecing cuts
each large periodic structure into spatial voxel pieces and runs selection over
pieces while keeping descriptors computed on the intact cells.

Use piecing when:

- The library has few huge structures, not many ordinary structures.
- A non-pieced run selects one or very few structures because each structure
  already contains all local states.
- Per-element group counts collapse, especially for minority elements.

Do not use piecing for a normal molecular/surface-adsorbate library with many
separate structures, and do not combine it with `loop.run: true` or
`loop.resume_from_iter`. Tune `piecing.target_atoms` by checking per-element
group counts at c*: lower `target_atoms` until each focus element resolves more
than one meaningful local state.

If `descriptor.kind: eds`, the default ED-SOAP settings are meant to run
immediately. The one value worth checking early is `descriptor.eds.lengthscale`:
set it near a typical nearest-neighbour bond length in Å for your material. Keep
`descriptor.eds.periodic: true` for crystals and slabs; set it to `false` for
isolated molecules or clusters. If `descriptor.kind: ace`, set
`descriptor.ace.model_path` to the MACE model file and choose `device: cpu` or
`cuda`.

Run once before tuning:

```bash
atom-hifi run config.yaml 2>&1 | tee run.out
```

For a long run, launch with unbuffered output and record the process id:

```bash
setsid stdbuf -oL -eL atom-hifi run config.yaml > run.out 2>&1 & echo $! > run.pid
```

Check output artifacts first, then the log. Output files are more reliable
completion signals than quiet stretches in the console output:

```bash
cat run.pid
ls -lh fr_results_NiO
test -s fr_results_NiO/representatives.xyz && echo "final representatives written"
tail -f run.out
```

For a standard run, expect these milestones in `paths.output_dir`:

| Artifact | Meaning |
|---|---|
| `desc_lib.pkl` | Structure descriptors were computed or loaded from cache |
| `eps_noise_raw.npz` | ε_noise calibration finished, unless the chosen path skips it |
| `fine_scan.out` | Final L/R scan table was written |
| `hifi_final.png` | Final scan diagnostic plot was written |
| `representatives.xyz` | Final representative structures were selected |
| `judge_final.pkl` | The fitted `Judge` was saved (unless `output.save_judge: false`) |

Optional stages add their own completion signals: `learning_curve.png` and
`descp_iter*.npy` for `loop.run: true`, `pareto_front.csv` and Pareto plots for
`nsga2.run: true`, and `surroundings*.xyz` when
`output.extract_surroundings: true`.

Common failures usually point back to one config group:

| Symptom | Check |
|---|---|
| `Unknown config key` | The key is not part of the YAML surface; remove it or use the Python API for expert-only controls |
| Input file error | `paths.lib_path` must point to an ASE-readable library from the run directory |
| Missing QHO-table element | Use `eps_noise.thermal.sigma_source: mass_only` or set `eps_noise.thermal.sigma_override` for that element |
| Out-of-memory or killed process | Check memory before reducing workers; descriptor work follows `compute.n_jobs_desc`, ε_noise calibration follows `compute.n_jobs_eps` or top-level `n_jobs` |
| Cache unreadable or incompatible | Atom-HiFi logs the cache problem and recomputes; keep caches only when descriptor, backend, and ε_noise settings match |

For follow-up runs, keep the same `paths.output_dir`: descriptor and
ε_noise caches are discovered there automatically when compatible. Use
`scan.cache_file` only for a compatible prior `el_fit_cache.npz` from the same
descriptor/backend/focus-element setup. Use `loop.resume_from_iter` only for
active-learning loop checkpoints; it does not resume an arbitrary interrupted
final scan. Current checkpoints include `descp_iterNN_source_idx.npy` and
`descp_iterNN_source_hash.npy`, which record and verify the exact ordered
source structures (including OOD-ranked batches); legacy checkpoints without
both files must be regenerated rather than
reconstructed by random-sampler replay.

## Starter template

Generate a commented `config.yaml` with:

```bash
atom-hifi init            # writes ./config.yaml
atom-hifi init my.yaml    # or a chosen path
```

The template below is emitted by `default_config_yaml()` and documents every
option inline.

```{eval-rst}
.. autofunction:: atom_hifi.config.default_config_yaml
   :no-index:
```

## Loading and validation

```{eval-rst}
.. autofunction:: atom_hifi.config.load_config
   :no-index:

.. autofunction:: atom_hifi.config.fill_defaults
   :no-index:
```

`load_config()` rejects YAML keys absent from the starter template, including the
API-only controls `grid`, `patient`, `paths.patient_path`, and
`refit.grid_point`. `fill_defaults()` is the Python API path and supplies hidden
defaults for those controls before {py:func}`atom_hifi.runner.run` consumes the
config.

## Custom descriptor

`descriptor.kind` accepts `eds` (ED-SOAP) or `ace` (MACE ACE) in YAML. A custom
descriptor is a Python callable and cannot be expressed in YAML — call
{py:func}`atom_hifi.runner.run` from Python with
`config['descriptor']['custom_fn']` set to `fn(List[Atoms]) -> ndarray`.