The Runner section (infinite swapping)

The [runner] section sets the workers of the sampler that every path-sampling run (tis, retis, explore, pptis, repptis) runs on: the asynchronous, parallel reformulation of replica exchange (infinite swapping) described in the moves guide. The other tasks (md, md-nve, md-flux, umbrellawindow and make-tis-files) take no [runner] section: their input stops with a ValueError before the run when it has one, so remove it.

This page documents which runs use the sampler, the keywords of the [runner] section, and the environment variable that sets the number of workers. For a worked example see the infinite-swapping tutorial and the runnable inputs under the per-engine examples/tutorials/path_sampling/<engine>/<system>/infinite_swapping/ cases.

Note

A complete input is shipped at examples/tutorials/path_sampling/turtlemd/1D-double-well/infinite_swapping/infswap.toml; run it with pyretis run -i infswap.toml. An input in the deprecated .rst format runs on the same sampler and takes the same keywords in a Runner section.

Runs on the sampler

Every path-sampling run is driven through the single pyretis run command, and every one of them runs on the infinite-swapping scheduler: a central coordinator hands work to a pool of parallel workers. That holds for every input with task = "tis", "retis", "explore", "pptis" or "repptis", and for every engine. Without a [runner] section the scheduler runs one worker. task = "infinite_swapping" and "infswap" belong to the deprecated legacy-runner schema (see the migration guide); an input of this version that sets them stops with a ValueError: set the path-sampling task instead.

An input that asks for a combination the scheduler does not support is refused with an error; it is not run another way. From Python, a run started with pyretis run can be continued cycle by cycle with pyretis.simulation.scheduler.scheduler_cycles(), given the configuration pyretis.simulation.setup.setup_config() reads from its output.toml; the script examples/tutorials/other/1D-retis-movie/retis_step_by_step.py shows how. The number of workers can be set outside the input with the environment variable below.

Keywords of the runner section

Example Runner section:

[runner]
workers = 4

This requests four parallel workers.

Table 37 Keywords for the runner section.

Keyword	Description
workers	Number of parallel workers in the scheduler pool.
files	Files pyretis tools init stages for each step.
wmdrun	GROMACS: the mdrun command of each worker.

Keyword workers

workers = integer

The number of parallel workers the scheduler drives, an integer of 1 or more; another value, such as 1.5, 0 or true, is refused, and so is a count below 1 in PYRETIS_WORKERS. Each worker propagates one trajectory at a time and is handed new work as soon as it frees up. The coordinator takes the finished moves back one at a time and, where the engine can, recomputes the energies of every new path from its stored frames, so the wall-clock time falls close to linearly with the number of workers only while the coordinator spends little time on a move against the time a move takes; with the internal engines it spends about as long, which limits the gain from more workers. workers cannot exceed the number of live path ensembles: pyretis run and pyretis tools init refuse a run with more before it writes a file. A move holds its ensembles until it finishes, and a [0^-]/[0^+] swap or an ∞REPPTIS swap holds two, so the moves in flight can hold every ensemble while a worker has no move: that worker waits until a move finishes and frees an ensemble.

In a run that swaps paths between ensembles, set workers to at most the number of live path ensembles minus one. With as many workers as live ensembles, every pick after the first moves finds one free ensemble and one free path, so the run swaps no path: in a retis run, infinite swapping exchanges no path between ensembles and the run samples as replica exchange without swaps; in a repptis run, the swap with a neighbouring ensemble finds that ensemble busy and the run samples as PPTIS. Such a run continues, with a warning in the log that gives the numbers. The tis, pptis and explore runs, a run of one ensemble and a repptis run with [retis] swapfreq = 0 swap no paths between ensembles: they can use as many workers as live ensembles and are not warned.

A worker is an independent Python process with its own molecular- dynamics engine. The coordinator chooses an ensemble and one of the paths it holds, gives that trial to a free worker, and incorporates the returned accepted or rejected path. It is therefore closer to a laboratory assistant completing one trajectory assignment at a time than to a thread splitting a single trajectory. For an external engine, each worker launches its own LAMMPS / GROMACS / CP2K process; any MPI ranks or OpenMP threads used inside that engine are a separate level of parallelism and count in addition to the worker pool.

With pyretis run -p, the Monte Carlo bar reports completed sampling cycles while its live postfix reports the most recently advancing worker, ensemble, propagation leg and leg length, for example worker 2 | ensemble 004 | forward leg length 17 | cap <= 2000. maxlength is displayed as a cap rather than a percentage because a path normally stops at an interface before reaching it and regular variable-length shooting may draw a still smaller stochastic cap.

With several workers, the scheduler takes the finished moves back one at a time, in the order their results reach it. While it treats one move (taking in its accepted path or keeping the old one, updating the occupancies, writing its output and picking the next move), a move that finishes meanwhile waits with its ensembles locked. The time at which its paths return to the free ensembles then depends on the durations of the moves and, through them, on the paths, and the sampled occupancies deviate from the product measure. In small models solved exactly, the deviation is small when the coordinator spends little time on a move against the time a move takes, largest when the two times are comparable, and smaller again when the coordinator is much slower. The internal-engine tutorials run near a ratio of 1, the two turtlemd infinite-swapping tutorials at about 0.06 and 0.2. With two workers at most one finished move waits while the coordinator treats another; with three or more, several can. A run with one worker has none of this.

A run with more than one worker measures this and reports it in the run log at its end, also when it is interrupted, from the moves taken so far. The coordinator reads a monotonic clock when it submits a move, when it asks for the next finished move and when it takes one; the worker reads the same clock at the moment it ends a move, and that is the finish time of the move. A move waited when it finished before the coordinator asked for it; its wait lasts from its finish to its take. The busy time \(L\) of a treatment runs from the take of a move to the next ask, and \(T\) is the time a move takes from its submission to its finish. The log line gives the number of moves, the percentage that waited, their mean wait, the mean \(L\), the mean \(T\) and \(L/T\), as a warning when more than 1 % of the moves waited. In the exact two-worker models with four classes of path, of move times 1:1:2:4 and 1:2:4:8, the deviation of the averages from the product measure is up to 2.5 % and 3.2 % at \(L\) comparable to \(T\), and is 0 at \(L\) negligible against \(T\). The times appear in the log alone.

With a warning, the sampled occupancies can deviate from the product measure at the percent level. A run with workers = 1 has no waits. With several workers, moves that are long against \(L\) keep the waits rare, and a run with workers = 1 or with another number of workers shows whether the rate moves beyond its error.

Default

The default is workers = 1 (a single-worker run, which reproduces the canonical RETIS loop).

Keyword files

files = list of strings

pyretis tools init (the automatic interface placement) copies these files, resolved relative to the input file, under their file names into the directory of each of its short runs, the working directory of the run. Each run imports the modules of the input, such as that of the orderparameter section, from the paths that pyretis run imports them from, or else from the folder of the input file, so a module finds the files next to itself; list here a file that a module opens by its file name alone, e.g. files = ["parameters.dat"]. A name that gives no file, and two files of one file name, stop pyretis tools init before its first run. pyretis run stops on it, as on an unknown keyword of [runner]: give it in the input of pyretis tools init only.

Default

The default is an empty list (no auxiliary files are copied).

Keyword wmdrun

wmdrun = list of strings

For the GROMACS engines (class = "gromacs" and "gromacs_steps"), the mdrun command of each worker, which takes the place of mdrun of the engine section in the moves that worker runs, so different workers can run on different devices or with different launch wrappers. Give one command per worker: the first command for the first worker, the second for the second, and so on; a list with fewer commands than workers stops the run with a ValueError. An engine class that a module derives from these engines is a GROMACS engine here too.

The kick that generates the initial paths runs mdrun, with or without wmdrun, so set mdrun as well. The sequential kick runs [engine] in the PyRETIS process, and the parallel kick (kick-parallel = true with kick-from = initial) runs, in the job of each ensemble, the engine section of the ensemble’s list in [simulation] ensemble_engines. A kick that runs a GROMACS section without mdrun is refused before it starts, and so is an input without wmdrun with a GROMACS section without mdrun.

With every other engine, wmdrun takes no effect: the workers of a LAMMPS or CP2K engine run the command of the engine section (lmp, cp2k), and the AMS, ASE, TurtleMD, OpenMM and internal engines launch no command. Leave it out for them, and set the launch command of LAMMPS and CP2K in the engine section.

Default

The default is unset: every worker runs the mdrun of the engine section of its move.

Setting the number of workers outside the input

PYRETIS_WORKERS=N in the environment sets the number of workers of every path-sampling run started there, in place of [runner] workers, so a job script can set it without editing the input; PYRETIS_NATIVE_WORKERS, its earlier name, is read when PYRETIS_WORKERS is not set. output.toml records the number of workers the run ran with as [runner] workers, and where it was read from as [current] workers_source: "PYRETIS_WORKERS", "PYRETIS_NATIVE_WORKERS" or "runner.workers" (the input). A resume of output.toml without the variable runs with the recorded number. The number of workers changes what the run samples (see workers), so give a restart or a resume the same value; the resume checks do not compare it.

Note

A run with mirror or target moves ([tis] mirror_freq or target_freq above 0, as in the permeability tutorial), and a biretis run, run with one worker: with more, also through PYRETIS_WORKERS, they stop with a ValueError before they start.

Output and analysis

Every path-sampling run writes the per-ensemble pathensemble.txt / order.txt / energy.txt files (in the numbered ensemble directories under the output data directory), which pyretis analyse and pyvisa read. [output] native_compat, a key of earlier versions, is refused as an unknown keyword: remove it.

Because the trajectories are shared fractionally across ensembles rather than collected from fixed, independent ones, the per-interface crossing probabilities and the final rate constant can also be combined with the Weighted Histogram Analysis Method (WHAM) rather than by simple per-ensemble averaging. WHAM reads the same per-ensemble output; a run writes no separate infswap_data.txt. See the analysis guide for how to run the WHAM crossing-probability analysis.