Trustworthiness workplan¶

Last reviewed:: 27 July 2026
Scope:: numerical correctness, reproducibility, reference integrity, and release evidence

Purpose¶

A green test is evidence only when the test demonstrably inspected the quantity it claims to validate. This workplan turns that rule into executable gates. It complements Method-equivalence validation: the double-well on one engine, Cross-engine validation: water-dimer dissociation, and the adversarial comparison audit in docs/test_integrity_audit.md.

Status terms¶

DONE: Implemented and exercised by an automated test or a reproducible command.
IN PROGRESS: Work is being implemented in the current change set.
BLOCKED: The code-side preparation is complete, but reference-grade software, hardware, or a deliberate reference re-baseline is still required.
TODO: Required work that has not started.

Definition of trustworthy¶

A release is trustworthy only when all of the following are true:

required unit, integration, method-validation, and external-engine jobs pass;
no comparison passes after inspecting zero rows, zero paths, zero swaps, an empty archive, or an unexpected all-NaN physical quantity;
every numerical reference records its generating engine, version, execution mode, command, tolerance rationale, and content digest;
each central analysis method has an analytical, hand-computed, or independently implemented oracle;
deterministic and stochastic workflows have separate contracts;
restart, load, interruption, and parallel shutdown behavior are tested as distinct semantics;
a file that cannot be read or written stops the run, instead of being absorbed into an empty read or a truncated output that still reports success;
reading a file gives the same result on every machine, independent of the locale;
running the tests does not modify the working tree, so no test can quietly rewrite a committed reference or input;
the release pipeline cannot publish when a required validation was skipped or downgraded to execution-smoke coverage.

Workstreams¶

W0. Integrate concurrent review work¶

DONE: Review the concurrent agent’s path-validation, shooting-weight, documentation, and changelog changes without reverting or duplicating them. Focused path-analysis and replica-exchange tests pass.

W1. Honest engine-suite outcomes¶

DONE: Engine absence must return the conventional skip status (77), and the orchestrator must report VALIDATED, SMOKE, SKIPPED, and FAILED separately.
DONE: Add PYRETIS_REQUIRE_VALIDATION=1. In this mode, a skipped or smoke-only required engine is a failing release gate rather than a green result.
DONE: Exercise both permissive developer mode and strict release mode with automated driver-contract tests.

W2. Reference integrity and physical coverage¶

DONE: Add a machine-readable provenance manifest for external-engine references, including deterministic tree digests and explicit tolerance rationale.
DONE: Require suites to declare energy terms that are intentionally unavailable. Unexpected all-NaN columns must fail, while declared unavailable columns must be checked to remain all-NaN on both sides.
DONE: Report finite OpenMM potential energy from the OpenMM context and regenerate its golden data twice with the pinned OpenMM Reference platform. The two runs were byte-identical, and the release-grade suite now requires the generated files to remain byte-identical.
DONE: Establish a run-to-run deterministic CP2K reference build: the conda-forge cp2k 2026.1 package (single rank, OMP_NUM_THREADS=1) reproduces the test-retis-load run byte-identically across clean runs (verified 2026-07-19); the golden was re-blessed on it and the suite validates only on that exact version/revision.
DONE: Re-baseline the legacy GROMACS 5.1.4 tarballs onto the single current reference build. The whole GROMACS reference suite now targets one era (double-precision 2025.4 on the development reference box, re-blessed 2026-07-19 from a determinism-checked double run), so strict full-scope validation is satisfiable by one executable.
DONE: Keep the test run out of the committed fixtures. Unit tests built engines against tracked input directories, so a run wrote pyretis.mdp and its logs into directories holding committed references, and the example suites left engine scratch in the shared input directory. Tests now work on copies under tmp_path (the example suites keep running in place and sweep with pyretis tools clean, since their goldens sit alongside). A full ./test-easy.sh now leaves the working tree byte-for-byte unchanged, verified by snapshotting every file before and after, so no test can quietly rewrite a reference or an input.
DONE: Calibrate each external-engine tolerance from repeated runs on its pinned reference environment. Two strict release-gate runs on 2026-07-27 (PYRETIS_REQUIRE_VALIDATION=1 PYRETIS_GROMACS_SCOPE=full, single rank, OMP_NUM_THREADS=1) both reported six-of-six VALIDATED, and every external-engine golden was reproduced with zero observed deviation across those runs and the 2026-07-19 blessing: the byte-exact GROMACS, LAMMPS, and OpenMM gates and the PyVisA content hash pass only on byte or hash identity, and the pinned CP2K build reproduces its golden byte-identically. The operative tolerance is therefore byte identity (measured envelope 0.0); the retained numeric diagnostics (CP2K 1e-4, OpenMM 1e-5) sit orders of magnitude above the measured envelope, and the provenance manifest records the measurement per set.

W3. Independent scientific oracles¶

DONE: REPPTIS cutoff occupancy is checked from raw columns with hand-audited counts and two independent probability constructions.
DONE: WHAM has hand-built synthetic exact-value tests, including weighted cases.
DONE: The method-validation campaign compares rate estimates with the analytical Kramers/Mel’nikov-Meshkov result.
DONE: Keep an evidence matrix that maps each public analysis result to at least one independent oracle and names any remaining unsupported result.

W4. Stochastic and parallel contracts¶

DONE: Single-worker seeded workflows retain byte-reproducibility checks; multi-worker workflows are explicitly completion-ordered and are not falsely required to be byte-identical.
DONE: A swap-consistency check that examines no swaps must report zero coverage and fail when the caller explicitly requested swap validation.
DONE: Promote the multi-seed method-validation campaign to a scheduled CI job with confidence intervals and stored provenance.

W5. Failure and recovery behavior¶

DONE: Async worker exception recovery, burst submission, forced cancellation, repeated lifecycle cleanup, SIGTERM unwind, and atomic continuation writes have automated coverage.
DONE: Add corruption tests for the reference manifest and explicit failures for missing or changed reference artifacts.
DONE: Add controlled disk-write failure injection around run-state persistence and verify that the last durable state remains restartable.
DONE: Surface file-I/O failures instead of absorbing them. FileIO used to log a failed open and keep a None handle, which iterates as end-of-file, so an unreadable or missing file was indistinguishable from an empty one; a failed write was logged and reported through a status value no caller reads, leaving a truncated energy.txt or order.txt behind a run that reported success. Both now raise. The path-ensemble reader used by pyretis analyse opened the file itself and swallowed a read error into an empty path set, so an unreadable or mid-read-truncated pathensemble.txt silently produced a wrong crossing probability; it now propagates the error too. Callers for which a missing file is a legitimate state check for it first and say so: a path loaded from sparse or external files carries no energies, and that is reported per path while the simulation continues.
DONE: Pin text decoding to UTF-8. The encoding was selected by an inverted test, so every text file was decoded with the platform locale, making a read depend on the machine it ran on. The on-the-fly trajectory reader shared by the streaming LAMMPS and CP2K engines and the LAMMPS dump writer also opened text without an explicit encoding; both now pin UTF-8, so no text open decodes or encodes through the locale. The remaining swallowed I/O error is the directory fsync in inout/common.py, which weakens durability without corrupting data and is documented as such.

W6. CI and release policy¶

DONE: Run reference-manifest validation and strict Sphinx warnings-as-errors in normal CI.
DONE: Make tagged releases depend on the strict validation jobs. Manual developer smoke jobs remain useful, but cannot satisfy a release dependency.
DONE: Coverage is collected on CI Python 3.12, separately from the Python 3.14/NumPy combination whose tracer is currently incompatible.

Execution gates¶

The intended commands, from least to most expensive, are:

python -m pyretis.testing.reference_manifest \
    examples/tests/reference_provenance.json
sphinx-build -W --keep-going -E -b html docs /tmp/pyretis-docs
./test-easy.sh
./test-heavy.sh
PYRETIS_REQUIRE_VALIDATION=1 ./test-heavy.sh

In permissive mode, missing or non-reference engines are reported as SKIPPED or SMOKE and do not by themselves fail the driver. The final command is the release gate and may not pass until every required engine is installed and reference-grade.

Current blockers and decision points¶

The CP2K determinism blocker is resolved (conda-forge 2026.1, single rank, deterministic; golden re-blessed 2026-07-19). Non-reference CP2K builds still run execution-smoke only.
GROMACS references are unified on a single era (double-precision 2025.4, the development reference box, re-blessed 2026-07-19): the former 5.1.4 archive suite was regenerated on the chosen reference version.
External-engine tolerances are now measured, not provisional: repeated strict release-gate runs on the pinned reference environment (the 2026-07-19 blessing and two runs on 2026-07-27) reproduced every golden with zero observed deviation, so byte or hash identity is the operative tolerance and the retained numeric diagnostics sit far above the measured 0.0 envelope.
Reference updates are review events. A digest update must include the engine provenance, generation command, observed error envelope, and reviewer signoff in the merge request.