Overview¶
The page below is the project README.md, included verbatim so it stays in sync
with the repository root.
Atom-HiFi¶
Atomistic High-Fidelity representative-set selection framework.
Applications include:
MLIP training-set curation and active-learning loops
Chemical motif identification and distribution analysis
Diversity-aware structure sampling from large databases
What is Atom-HiFi?¶
Atom-HiFi finds the smallest subset S of a structure library that achieves high Fidelity — meaning S covers the library’s atomic-environment diversity efficiently, without redundancy. Agnostic to the downstream task.
Key concepts¶
Fidelity = L / R¶
Fidelity is the single optimisation objective. Like a HiFi audio system, it has two channels — L (Left) and R (Right) — whose ratio is maximised. High Fidelity means the selection is both faithful to the library distribution (high Likeness) and compact (low Redundancy).
L — Likeness measures how faithfully S reproduces the library’s
atomic-environment distribution. Each focus atom is assigned to a
microstate: a per-element k-means cell in embedded descriptor space. The
default granularity is conservative, k = N//2 per focus element. Known-rigid
systems may opt into the ε_noise-based r100 rule with
microstate.reference: r100. Training-set studies may opt into the force
resolution rule microstate.reference: f_tol, which derives ε_F from the
cached thermal ε(σ) curve and eps_noise.force.f_tol; L is the Shannon entropy
ratio over those populations:
L = H(sub) / H(lib) H = -Σ p_i ln p_i
Shannon entropy H measures distributional diversity — how evenly the population is spread across microstates. L = 1: S perfectly reproduces the library’s diversity. L < 1: some environments are under-represented; e.g. L = 0.95 means S retains 95% of the library’s distributional diversity.
R — Redundancy measures how many atoms are packed per occupied microstate, relative to the full library:
R = (N_sub / k_occ^sub) / (N_lib / k_occ^lib)
R = 1: same atoms-per-microstate density as the full library (no compression). R < 1: redundancy has been removed; e.g. R = 0.4 means 60% of redundant atoms are eliminated while the occupied microstate coverage is preserved.
The scan sweeps a bandwidth c (scaling factor on ε_noise) and finds c*
that maximises Fidelity subject to L ≥ scan.l_tol (default 0.95).
The optimal c* sits at the elbow of the L/R curve — the point where further
reducing redundancy begins to cost meaningful distributional diversity.
ED-SOAP descriptor¶
Embedded Double SOAP — two concatenated SOAP power-spectrum vectors per atom: one short-range
(bonding geometry) and one long-range (coordination shell), normalised by a
system-specific lengthscale. No GPU required. The full parameter set is
exposed in hifi_workflow_tutorial.py under the EDS_* variables.
Installation¶
pip install atom-hifi
Python ≥ 3.9 required.
Documentation¶
The Sphinx documentation lives in docs/sphinx/ and includes this README plus
autodoc API pages. Build it locally with:
pip install -e ".[docs]"
sphinx-build -b html docs/sphinx docs/sphinx/_build/html
The GitLab Pages job publishes the same site. It uses the Furo theme; set
ATOM_HIFI_DOCS_MAIN_COLOR=#0057b8 (or another CSS hex color) to change the
brand color in CI or local builds.
An experimental, opt-in labels-first temporal assessment
(python -m atom_hifi.experimental.temporal) is documented in
docs/sphinx/experimental_temporal.md; it changes no default and its B/E ratio is not
the selection fidelity L/R.
Quick start¶
pip install atom-hifi installs the atom-hifi command. Write a starter config,
edit it, and run:
atom-hifi init # writes a commented config.yaml
# edit config.yaml (at minimum: paths.lib_path, paths.focus_elements)
atom-hifi run config.yaml 2>&1 | tee run.out
# optional for microstate.reference: f_tol
atom-hifi run config.yaml --f-tol 0.08
At the default log level a run prints stage summaries plus a live progress bar
for each long loop (1-D line scan, NSGA-II). Redirected or tee
logs keep checkable progress lines, so tail -f run.out shows scan/optimizer
movement. Use -v for full debug detail or -q for warnings only (or set
log_level: debug|info|warning|error in the config).
The generated config.yaml documents every setting inline. The minimum to edit:
paths:
lib_path: train_structs.xyz # ASE-readable structure library
focus_elements: [Ni, O] # elements to cluster on
output_dir: fr_results
descriptor:
kind: eds # 'eds' or 'ace'
Clusterer backends¶
cluster.backend selects the clustering algorithm (default decaf = mean-shift,
unchanged behaviour). kmeans is the fastest selector with the smallest covers — it
scans cluster counts (a k-fraction axis) instead of the c·ε_noise bandwidth, and skips
ε_noise calibration under the default N//2 microstate reference. hdbscan keeps the
c·ε_noise sweep and adds an honest noise channel: each distinct noise environment
(deduplicated at radius ε_noise) forces its structure into the representative set and
gets its own coverable group and surroundings sphere (noise_derived=True).
All three built-in backends support OOD/patient classification and the
active-learning loop. HDBSCAN uses frozen density hierarchy attachment;
promoted noise groups are selection representatives, not deployment targets.
See HDBSCAN deployment for rejection
semantics, persistence, and a runnable saved-Judge example. YAML
exposes the loop only, while patient/OOD remains available through the Python API.
Because kmeans’ resolution-tracking OOD can remain positive as k grows, its loop may
consume the full library and warns once when started. The config rejects invalid
backend combinations.
Scan caches are backend-tagged and cannot be reused across backends.
Reuse also requires matching ordered corpus, numeric descriptor rows/mapping,
backend settings and actual fitted granularity; a distance cover also serves a
request for its own 10-decimal cache key, as on the grid’s default and coarse
axes (the fitted c rounds to that key, and the grid row reports δ = key·ε).
Distance-backend covers remain reusable when only the microstate scoring
partition changes; count-backend covers must match the resolved cluster count.
Legacy covers without input/granularity evidence are readable but recomputed
before use.
Custom backend identities include configured instance/class data and inherited
private slots; registered classes use their default constructor. Opaque settings
prevent cover reuse and snapshot restoration, but a completed scan’s axis values
remain available for a default grid whose covers are recomputed.
Python API / custom descriptors¶
The CLI supports the eds and ace descriptors. A custom descriptor is a
Python callable and is supplied via the Python API. hifi_workflow_tutorial.py
is the annotated example (included in the repo; pip-only users can fetch it):
curl -O https://gitlab.mpcdf.mpg.de/yhsong/atom-hifi/-/raw/main/hifi_workflow_tutorial.py
Edit its top-level variables (including DESCRIPTOR_FN) and run
python hifi_workflow_tutorial.py, or call the runner directly:
from atom_hifi.runner import run
run({'paths': {'lib_path': 'train_structs.xyz', 'focus_elements': ['Ni', 'O']},
'descriptor': {'kind': 'custom', 'custom_fn': my_descriptor_fn}})
Thermal ε-noise calibration keeps only per-rattle mean/std summaries in
HiFiWorkflow.eps_results by default. Raw samples arrays are not used by later
workflow stages and can be large. Python callers that need the historical full
result shape for diagnostics can construct the workflow with
retain_eps_samples=True, or call
atom_hifi.training.judge.measure_eps_noise directly.
Output files¶
File |
Description |
|---|---|
|
Selected representative structures; with piecing, unique full parents with every focus atom tagged |
|
L, R, F (=L/R), |S|, atoms for every fine-scan point |
|
The fitted |
|
Coarse + fine Fidelity (F = L/R) scan diagnostic plot |
|
AL loop convergence (only with |
|
Cached per-element ε_noise values and, for thermal calibration, ε(σ) curves |
|
Cached per-structure descriptors |
|
Cached per-structure patient descriptors ( |
|
Descriptor input identity and cache-content fingerprint |
|
Per-group coordination spheres ( |
|
Refined/sorted group-id table: element, backend-native → sorted local → global |
|
Per-element center linkage + leaf order (post-hoc meta-group recuts) |
|
Per-atom |
|
ND grid-scan results (Python API |
|
NSGA-II front; |
|
NSGA-II L-vs-R front; F-vs-log|S| and F-vs-log(atoms in S) fidelity views |
|
NSGA-II parallel-coordinates + HV convergence |
Descriptor pickle files retain their list-of-arrays format. Built-in descriptors are reused only when a companion manifest matches the settings, ordered structure contents (including positions, atomic numbers, cell, periodicity, arrays and info), and pickle contents. ACE also binds the contents of a readable local model file. Legacy caches without manifests, damaged or mismatched pairs, and inputs or models that cannot be fingerprinted are recomputed. Custom Python callbacks are always recomputed because their captured or global state may change between calls.
Automatically calibrated eps_noise_raw.npz values and thermal ε(σ) curves also
bind this descriptor/corpus/model identity, focus species, calibration sampling
seed and sample size, and effective calibration settings. Matching inputs reuse
both descriptors and calibration; changed or unverifiable inputs recalibrate.
Legacy raw NPZ files retain their readable fields, but the runner recalibrates
files lacking the new input binding. Explicit eps_noise.method: precomputed
values remain authoritative. Changing force tolerance can reuse a matching full
thermal curve because the tolerance is applied after calibration.
Loop resume verifies saved descriptor values and row ordering against the current
recorded sources. Refit (refit.delta or Python refit.grid_point) intentionally
uses the snapshot’s saved ε and microstate reference at a new operating point;
it verifies the corpus, actual descriptor values/mapping, backend settings and
reference integrity before attaching that state. Changed descriptor inputs or
unverifiable legacy snapshots require a new scan. Presentation settings and a new
microstate-reference request do not replace the authoritative saved reference.
Final-fit outputs use refined/sorted group ids (issue #40): per element, group
centers are ordered by average linkage + optimal leaf ordering so adjacent ids are
nearby local atomic states, then offset by species order so group_id is globally
unique across elements (noise stays -1). A meta_group_id marks contiguous id
blocks that form separated macro basins (gap-statistic gate + silhouette-selected
dendrogram cut; single meta-group when no macro structure exists). The relabel is a
bijection applied only to the chosen final fit — scans, caches and operating-point
selection keep backend-native labels, and |S|/L/R/F are unchanged. Ids are a
within-fit inspection aid, not stable identities across runs/backends/datasets.
The final-fit PNGs (final_fit_delta{δ}_{el}.png, final_fit_grid_*_{el}.png)
color the PCA panel by meta_group_id with group centers marked, so the macro
basins are visible directly in the per-element diagnostic (issue #43).
Configuration reference¶
All settings live in config.yaml (run atom-hifi init to generate a fully
commented template). Keys are grouped:
Group |
Keys |
|---|---|
top-level |
|
paths |
|
descriptor |
|
selection |
|
scan |
|
microstate |
|
eps_noise |
|
loop / nsga2 |
|
refit |
|
output |
|
surroundings |
|
piecing |
|
Unknown YAML keys are rejected. The Python API accepts the same nested dict plus
expert-only controls for ND grid scan and patient/OOD:
grid, patient, paths.patient_path, and refit.grid_point.
hifi_workflow_tutorial.py is the annotated Python-API equivalent.
Advanced usage¶
Microstate reference (microstate.reference)
microstate.reference: n2 is the default: per focus element, Atom-HiFi uses an
independent k-means partition with k = N//2. This is the conservative,
system-transferable reference used by the L/R scan.
microstate.reference: r100 is opt-in for known-rigid systems. It chooses the
smallest resolved k whose KMeans cluster radius is within ε_noise, searching up
to N_atoms-1. A result finer than the n2 baseline is kept with a warning because
the increased singleton pressure may weaken L discrimination. If the ceiling
still fails, Atom-HiFi reports the thermal-reference undersampling ratio. The rule
can under-resolve soft or superionic sublattices, so use it only when that physical
assumption is valid.
microstate.reference: r<NUM> (NUM in 1–100) is the same ε-radius criterion
with a user-defined strictness: NUM is the within-cluster-distance percentile,
r100 being the maximum (identical to r100). Hazard: a thermal
r<NUM> below 100 is an explicit user opt-in — the historical r90 criterion
under-selected k ~3× with +56 % force error on PdMgO. When the config carries an
explicit eps_noise.force block (or --f-tol is passed), r<NUM> resolves
against ε_F like f_tol, overriding that mode’s internal p95 default.
microstate.reference: f_tol is an opt-in force-resolution reference. It uses
u* = F_tol / κ, with κ from the packaged QHO second-moment table, then
interpolates ε_F from the cached thermal ε(σ) curve. Configure
eps_noise.force.f_tol (default 0.05 eV/Å per atom), or override it at the CLI
with atom-hifi run config.yaml --f-tol 0.08. Like thermal adaptive references,
this mode is not capped at N//2; it can climb to N_atoms-1 when the requested
force resolution demands finer sampling. This mode requires
eps_noise.method: thermal; RDF and precomputed ε do not contain the ε(σ) curve
needed for ε_F (the same holds for a force-resolution r<NUM>).
For the durable rationale, see CONTEXT.md (Microstate (μ), Thermal truth,
Force truth) plus docs/adr/0003-scan-range-anchored-to-eps-noise.md and
docs/adr/0008-qho-sigma-table-default.md.
Active-learning loop (loop.run: true)
Iteratively expands the training pool by sampling batches from the full library.
Inner iterations use a coarse scan only; one final fine scan runs at the end.
Set loop.initial_sample and loop.skip_fine_scan to control the initial pool
size and inner-scan resolution.
Per-element ND grid scan (Python API only)
Sweeps independent c-factors per focus element on a Cartesian grid, reusing
cached per-element DECAF fits from the 1-D scan. Cost is O(n^N_el) cover
evaluations instead of O(n^N_el × N_el) DECAF fits — tractable for N_el ≤ 3–4.
Call atom_hifi.runner.run({'grid': {'run': True}, ...}) to enable it. Results
in scan_grid.csv and scan_grid_report.png.
NSGA-II Pareto optimisation (nsga2.run: true)
Stochastic multi-objective optimisation of per-element c-factors via NSGA-II
(requires pymoo). Use when the grid is too large (N_el ≥ 4) or you want a
continuous Pareto front. Outputs pareto_front.csv — with two operating points
tagged: selected (smallest R with L ≥ l_tol — the point that drives the
final fit; when l_tol: null, this is fidelity_best) and
fidelity_best (global argmax F = L/R, reported as a candidate).
Diagnostic plots: pareto_front.png (L vs R front),
pareto_fidelity.png (F vs log|S|) and pareto_fidelity_atoms.png
(F vs log(atoms in S)), plus pareto_parallel.png and
pareto_convergence.png.
RDF ε_noise calibration (eps_noise.method: rdf)
The default ε_noise calibration is thermal: rattle structures by an offline
Materials Project QHO σ_thermal table (thermal.sigma_source: qho_table) and
measure the induced descriptor shift. Use thermal.sigma_source: mass_only for
the legacy mass + temperature model, or thermal.sigma_override to pin exact
per-element σ values in Å for the current run temperature. Thermal calibration
also stores the ε(σ) curves used by microstate.reference: f_tol.
eps_noise.method: rdf instead estimates the displacement scale from the
database’s RDF first-peak width, then applies fixed-magnitude center-only
rattles to produce a per-element ε_noise.
RDF ε_noise always rescales the c-factor scan bandwidth. It only changes
microstate granularity when paired with an r… reference
(microstate.reference: r100 or r<NUM>); otherwise k_micro stays at the
default N//2. Exposed RDF knobs are
eps_noise.rdf.n_samples and eps_noise.rdf.eps_stat (mean, median,
q95). Elements without a usable RDF peak borrow the mean ε from peaked
elements and force k_micro back to N//2; if no element peaks, calibration raises
instead of silently falling back to thermal. The eps_noise_raw.npz cache is
fingerprinted by method, RDF variant, thermal σ source, and exact σ targets.
Representative environment extraction (output.extract_surroundings: true)
Exports the local coordination sphere around the centroid-closest atom of each
DECAF group. Two modes: 'sphere' (non-periodic ASE Atoms cluster) and
'full_structure' (original cell with center/neighbour/rest tags). Output:
surroundings_{el}.xyz per focus element.
Piecing for super-large structures (piecing.enabled: true)
When the library is a few very large structures (~1000 atoms), each one already
contains nearly every microstate, so set-cover returns |S| = 1 and the c-factor
inflates — the fidelity metrics saturate. Piecing cuts each large structure into
disjoint, spatially local voxel pieces (granularity set by target_atoms) and
runs selection over pieces instead.
It is a statistics-layer cut only: descriptors are computed once on the
intact periodic cell and pieces merely select rows, so chemical environments
stay exact and the clustering/scan stack is unchanged. After selection, per-atom
group ids are mapped back onto the original structures in diagnosed.xyz /
diagnosed.npz (motif_id, piece_id, in_selection). With piecing enabled
|S| and fidelity reports count pieces, while representatives.xyz contains
deduplicated full parent structures in first-selected-piece order. Every focus
atom in each representative carries its final group_id and meta_group_id,
including atoms from sibling pieces not selected by set-cover. Not supported
with the active-learning loop or resume_from_iter.
Tuning target_atoms. The right metric is the per-element DECAF group count
at c* (distinct local atomic states resolved), not |S|. Coarse pieces leave
c* high, so minority elements collapse to a single group (no diversity captured)
— the run logs a warning when any focus element resolves to ≤ 1 group. Lower
target_atoms until no element collapses. (On a PtOH test, oxygen resolved to 1
group at target_atoms 64/128 but 7 groups at 32.)
Voxel grid. Each structure is divided in fractional coordinates: target edge
L = (target_atoms / ρ)^(1/3) (ρ = N/V) determines the number of divisions
along each cell vector. Atoms in the same voxel form one piece. The grid is
PBC-correct and triclinic-safe; structures without a 3-D cell fall back to a
bounding-box grid. Empty voxels are dropped; sparse voxels are kept as-is. The
partition must be spatial — index-based chunking would put a uniform microstate
sample in every piece, recreating the saturation artefact.
Citation¶
If you use Atom-HiFi in your research, please cite:
[paper in preparation — citation will be added upon publication]