Cleaning run artifacts

A PyRETIS run leaves a number of generated files and directories behind – log files, the regenerated out.toml / out.rst, restart files, the NNN ensemble directories (each holding its generate scratch, the accepted operational trajectory store and the archive long-term store), the report directory of pyretis analyse, Python byte-code caches, and so on. The pyretis tools clean sub-command removes them so a run (or example) directory can be reset to its inputs (a load directory, the default [initial-path] load_folder a load reads its initial paths from, is an input and is never cleaned):

pyretis tools clean              # clean the current directory
pyretis tools clean some/dir     # clean another directory
pyretis tools clean --dry-run    # list what would be removed, delete nothing

pyretis tools clean deletes immediately; use --dry-run (or -n) first if you want to preview the list. A directory named runs is removed with the inputs in it, although PyRETIS does not write one: keep your runs in a directory of another name, or list runs under keep (below).

Note

pyretis clean (without tools) still works as a deprecated alias: it prints a one-line notice and then behaves identically. Use pyretis tools clean in new scripts.

What is removed

The removed set is the union of a comprehensive built-in default and an optional per-directory configuration file. The built-in defaults cover the files and directories pyretis run and pyretis analyse write, so a plain run folder resets with no clean.toml at all:

  • files – byte-code / compiled order parameters (*.pyc, *.pyo, *.so), the run file and restart store (output.toml*, out.toml*, out.rst*, restart.toml*, pyretis.restart, ensemble.restart), the per-run text output (energy.txt*, order.txt*, pathensemble.txt*, moves.txt*, trials.txt*, cross.txt*, thermo.txt*, traj.txt*), the logs of both pyretis run and pyretis analyse including their backup copies (pyretis.log*, pyretisanalyse.log*, sim.log, worker[0-9]*.log) and the engine logs (engine.log, engine.err), the starting frame a parallel kick writes in the run directory (kick_bootstrap.xyz), the start configuration a run writes for TurtleMD in the run directory (turtlemd_initial.xyz and turtlemd_initial_NNN.xyz), the engine scratch of every engine (*.tpr, mdout.mdp, pyretis.mdp, pyretis_md.*, order_pyretis_md.txt, *.edr, *.cpt, step.inp, continue.inp, *.lammpstrj, *.screen), the soft-exit file EXIT, and names some example and test scripts use (*-run.toml, energy.dat*, traj.gro*, rnd.state, compare.png). A file of your own whose name matches one of these patterns is removed as well – an input named my-run.toml, a structure named traj.gro – unless keep (below) lists it;

  • directories – __pycache__, the NNN ensemble directories ([0-9][0-9][0-9], which subsume their generate / accepted / archive / engine_logs sub-stores), the accepted / archive / engine_logs stores and the previous store names (paths / trajs), the retired worker* scratch, the report and runs directories, and the private directories a run removes once it is done with them, left behind by a run that stopped first: pyretis-gromacs-*, the work directory of a GROMACS engine, removed with the engine, and pyretis-load-*, where a load prepares its files, removed when the load ends;

  • the per-ensemble store of the paths of a path-sampling run, <ensemble>/<load_dir>, under the [simulation] load_dir a TOML input names (by default accepted), in each ensemble directory of that run: the directories named by ensemble number (000, 001, …, 1000, …) in the directory of the file, or in its [output] data_dir. The TOML files are read as for the load input of the runs (below), before any file is removed, so the output.toml of a run names its store too. The store is removed also where its ensemble directory is left, because it holds a kept path or has a name of four digits. A store reached through a symbolic link, or a load_dir that points out of the ensemble directory, is left, and use_defaults = false leaves every store.

Three mechanisms keep these otherwise-aggressive defaults away from inputs:

  • Excluded reference/input directories. Anything below results, output_data, load, load_copy or an engine input directory (lammps_input, gromacs_input, cp2k_input, openmm_input) is never cleaned. These hold committed reference (golden) output, committed load/NNN initial-path inputs and committed engine input files.

  • The load input of the runs. A TOML input file names the paths the run in its directory reads: the [initial-path] load_folder the input sets, and the flat store <load_dir> of the run directory, where initial paths can be staged ([simulation] load_dir, by default accepted), both relative to the directory of the file. The TOML files in the cleaned directory and in every directory below it are read, so a clean started above a run directory keeps them too. They are left with all their files, whatever the default patterns, find_files, find_dirs or rm_paths match; remove one by hand once you are done with it. A TOML file that cannot be parsed stops the clean before it removes anything. A load folder that only a legacy .rst input names is protected with keep.

  • The ``keep`` list (below), for committed inputs that live outside those excluded names – for example the load folder that only a legacy .rst input names.

pyretis tools clean does not consult git: a file under version control whose name matches a pattern is removed like any other, so run --dry-run first in a directory you track.

Per-directory configuration

Drop a clean.toml next to the files to extend (or replace) the defaults:

[clean]
use_defaults = true        # apply the built-in defaults (default: true)
find_files = ["out.toml*", "*.log"]   # extra file-name globs (recursive)
find_dirs  = ["report"]               # extra directory names (recursive)
rm_paths   = ["0*", "lammps/system.data"]  # root-relative rm -rf globs
keep       = ["pippo"]                # never delete these (or ancestors)

  • find_files – file-name globs deleted wherever they occur in the tree (like find . -name PATTERN -delete).

  • find_dirs – directory names deleted recursively wherever they occur.

  • rm_paths – root-relative paths or globs removed with rm -rf semantics.

  • keep – paths that must survive; a kept path, anything inside it, and – since keep declares a committed input – the declaration is honoured even when the clean was started from an ANCESTOR directory (its clean.toml is read during the walk), and its ancestor directories are never removed. Use this to protect committed inputs that would otherwise match (for example the load folder of a legacy .rst input).

  • use_defaults = false disables the built-in defaults so the file is the complete, explicit recipe.

This replaces the per-example Makefile clean targets used in earlier PyRETIS versions.