Simulation Factory Advanced Tutorial

The Simulation Factory simplifies many aspects of running Cactus-based simulations. It provides a central facility for managing authoritative source tree versions, providing convenient access to remote HPC systems, building Cactus source configurations, and managing simulations all the way from submission to archiving their output.

Getting Started

To begin using The Simulation Factory, it needs to be checked out from svn. The Simulation Factory is typically placed into a simfactory folder inside a Cactus source tree. This can be accomplished with the following svn command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location to be used with multiple Cactus source trees. This approach will be described later.

Initial Setup

Once the Simulation Factory has been checked out from svn, the next step is to configure it, telling it e.g. about your user name. The Simulation Factory comes with example configuration files that you can copy and modify. Assuming The Simulation Factory has been checked out into the simfactory folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit simfactory/etc/defs.local.ini and replace

• YOUR_LOGIN with your usual username
• YOUR@EMAIL.ADDRESS with your usual email address
• YOUR_ALLOCATION with your usual allocation

Additional Configurations

The Simulation Factory contains a database known as the Machine Database. This collection of information describes all the aspects that are unique about each individual HPC system, so that the Simulation Factory can provide a common interface for all systems that hides these differences.

The Machine Database consists of different sections, one for each machine. The section name is given in square brackets, e.g. [queenbee]. There is a special section [default] that provides default values for those properties that are not explicitly set in the machine-specific entries.

The Machine Database is an authoritative collection of information, and is generally not meant to contain modification that are only relevant for individual people. These local modifications are instead maintained in the file simfactory/etc/defs.local.ini described above, where one can add, change, or overwrite properties of Machine Database entries. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine queenbee, you would add the following section there:

[queenbee]
user          = QUEENBEE_USERNAME
allocation    = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries. The most useful is probably @USER@, which expands to the user property of the Machine Database entry.

For example, if you are using the same user name on many systems, but have a different user name on some systems, then you would set the common user name in the [default] section, and override this for those machines where your user name differs. The example simfactory/etc/defs.local.ini.complex has examples for this.

Most of the macros available in the Simulation Factory are described in the section #Macros below.

The command

simfactory/bin/sim list-machines

outputs a list of all preconfigured machines that the Simulation Factory knows about.

Local Workstation Configuration

The Simulation Factory can only be used on a machine known to the Simulation Factory. This means that you may have to add an entry for your workstation or notebook.

The first step is to determine the hostname of the local machine. It is through this hostname that the Simulation Factory matches a Machine Database entry to the machine on which it executes. The hostname can be determined using the following command:

hostname

Once you know the hostname, issue the following command to create a new Machine Database entry, starting from a generic example:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Then edit simfactory/etc/mdb/<hostname>.ini and replace

• [generic] with [<hostname>]
  • The section header for this machine database entry must be unique among all Machine Database entries, and must match the nickname property exactly.
• nickname = generic with nickname = <hostname>
• hostname = generic with hostname = <hostname>
• sourcebasedir = /home/@USER@ with the correct root path under which all your Cactus source trees reside. (This is not the Cactus directory itself, but the directory that contains the Cactus directory.)
• basedir = /home/@USER@/simulations with the desired folder that will contain all simulation output. (This is explained in more detail below in the section #Managing Simulations.)

You can ignore the properties user, email, and allocation, as the values from the [default] section of simfactory/etc/defs.local.ini will propagate to this entry.

Accessing Remote Systems

The Simulation Factory simplifies access to remote systems, both for transferring files and logging in. You can synchronise (replicate) an authoritative version of your Cactus source tree to remote systems, obtain an interactive shell, or execute commands.

Information Commands

The following commands can be used to discover information about a machine, or list all known machines.

List all known machines:

simfactory/bin/sim list-machines

List details about a single machine:

simfactory/bin/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/bin/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/bin/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is currently being executed:

simfactory/bin/sim print-machine

Syncing

Historically, Cactus and the Einstein Toolkit have not been installed into a central location on each machine, but are instead built on-demand by every user for a certain thorn list. (One of the advantages is that people can thus easily add their own thorns.) To help with this approach, the Simulation Factory provides a facility to synchronize a Cactus user's local, authoritative source tree to remote HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like mechanisms such as gsi-ssh. Currently, you must still manage all ssh keys and passwords manually. (We highly recommend to use ssh keychain and ssh agents to avoid having to enter passwords multiple times.)

Configuration

Before syncing a source tree to a remote system, a small amount of configuration must be performed. It is necessary to either verify that the defaults are correct, or to define the correct values for the following keys for the remote system in the Machine Database:

• sourcebasedir
  • The root directory under which the Cactus source tree will reside
• basedir
  • The root directory in which all simulation output will reside
• user
  • The user name on the remote system

You can output the currently configured values by issuing the command

simfactory/bin/sim print-mdb <machine>

If you need to change these values, then edit (on the local system) the file simfactory/etc/defs.local.ini and add a section for the remote machine. This entry will augment the existing Machine Database entry and updating/replace the corresponding values. An example for the machine queenbee can be see in the [[#Additional Configuration]] section.

To see the list of files and directories that are synchronized, look at simfactory/etc/defs.ini and find the following two keys

• sync-sources
  • The list of files and directories that will be transferred when the option --sync-sourcetree is enabled (on by default)
• sync-parfiles
  • The list of files and directories that will be copied when the option --sync-parfiles is enabled (also on by default). This list of files typically includes just parameter files.
• sync-excludes
  • A list of files and directories that will be expressly excluded from syncing, such as e.g. CVS or .svn directories.

Performing a Sync

A sync command takes two options, both of which default to true.

• sync-sourcetree
  • Synchronise the complete source tree, as specified in the aforementioned rsync-sources configuration entry. This takes a few seconds or minutes, depending on the connection.
• sync-parfiles
  • Synchronise parameter files, as specified by the aforementioned rsync-parfiles configuration entry. This is typically faster than synchronising the source tree.

Usually, you would issue the command:

simfactory/bin/sim sync <machine>

To synchronise only parfiles, you can negate the --sync-sourcetree argument with the following command

simfactory/bin/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from another remote machine, then use

simfactory/bin/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine <frommachine>.

Remote Login

The Simulation Factory provides the ability to log in to a remote system. This is initiated with the command

simfactory/bin/sim login <machine>

This will automatically cd into the Cactus directory on the remote system.

Local/Remote Command Execution

To execute a command (locally) via the Simulation Factory, use the

command

simfactory/bin/sim execute <command>

The command will be executed in the Cactus directory on the remote system.

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/bin/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/bin/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/bin/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

Build Cactus Configurations

The Simulation Factory provides a central facility for configuring and building Cactus source trees. When a Cactus source tree is compiled, the Simulation Factory creates a configuration for the compiled executable, storing with it related information such as the Cactus options list, and the scripts necessary to submit and run jobs in a queuing system. This configuration is thus a self-contained entity containing everything that is necessary to perform Cactus simulations.

Information Commands

To list all existing Cactus configurations, use the following command

simfactory/bin/sim list-configurations

Building a Configuration

To build a configuration, four pieces of information are required:

• Thorn list
  • This defines which thorns are to be included into the configuration.
  • Default: thornlist parameter of the Machine Database entry
  • Override: --thornlist=<thornlist>
  • The default option list is probably not useful in many cases.
• Option List
  • This specifies the compiler and build options that need to be used to build Cactus on a particular system.
  • Default: optionlist parameter of the Machine Database entry
  • Override: --optionlist=<optionlist>
  • The Simulation Factory is supposed to contain good, working default option lists for all supported systems. In fact, this is one of the main strengths of the Simulation Factory. You should normally not need to override the default.
• Submission Script
  • This specifies how to submit a job to the queueing system on a particular system.
  • Default: submitscript parameter of the Machine Database entry
  • Override: --submitscript=<submitscript>
  • Similar to the option list, the Simulation Factory is supposed to contain good, working default submission scripts for all supported systems.
• Run Script
  • This specifies to how execute an MPI process on a particular system; it is closely connected to the submission script.
  • Default: runscript parameter of the Machine Database entry
  • Override: --runscript=<runscript>
  • Same as with the the submission script, the Simulation Factory is supposed to contain good, working default run scripts for all supported systems.

To build a configuration with a specific thornlist, issue the following command:

simfactory/bin/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. (We recommend this.) You can in addition specify any of the options

• --debug
• --optimise (default)
• --profile

These options create configurations for debugging, optimisation (this is the default), or profiling enabled. If any of these options is specified, then the configuration name will be modified correspondingly, e.g. to 'sim-debug'.

Additional Options

• --reconfig
  • Reconfigure before building, i.e. re-examine the configuration options and re-run the CST stage. This happens automatically when the option list changes.
• --clean
  • Clean the configuration (remove all object files etc.) before building.

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

Script Locations

The Simulation Factory provides default scripts for all its preconfigured machines. These scripts can be found in the following locations

• Option Lists
  • MDB Key: optionlist
  • Location: simfactory/etc/optionlists
• Submit Scripts
  • MDB Key: submitscript
  • Location: simfactory/etc/submitscripts
• Run Scripts
  • MDB Key: runscript
  • Location: simfactory/etc/runscripts

To determine, for instance, which option list Queen Bee uses by default, issue the command

simfactory/bin/sim print-mdb queenbee | grep optionlist

Managing Simulations

The Simulation Factory provides a convenient, consistent facility for submitting, running, and managing simulations. This is accomplished through two main commands submit and run.

Information Commands

The status of all simulations on a particular machine can be seen with the following command

simfactory/bin/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/bin/sim list-simulations --verbose

Submitting a Simulation

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

• Configuration
  • The Cactus configuration to run
  • option: --configuration
  • default: "sim"
• Parfile
  • The Cactus parameter file to use
  • option: --parfile
• Walltime
  • The total amount of wall time required
  • option: --walltime
  • default: MDB Key maxwalltime
• Processors
  • The total number of processors to use
  • option: --procs
  • default: 1

The option --configuration only needs to be specified the first time you submit a simulation. Subsequent re-submissions of the same simulation (for restarting from checkpoints) will always use the same configuration that was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options:

simfactory/bin/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where you do not need to specify the option names, but have to specify the options in a certain order. If you don't specify a simulation name using the shorthand syntax, a simulation name will be derived from from the parameter file name.

simfactory/bin/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/bin/sim submit par/static_tov.par 4:00:00 8

Additional Options: Submission

• Number of OpenMP Threads
  • The number of OpenMP threads per MPI process. (You specify the total number of processors (cores), and the number of OpenMP threads; the number of MPI processes is then calculated automatically.)
  • option: --num-threads
  • default: 1 (as if OpenMP was not used)
• Allocation
  • The allocation for the simulation, overriding the corresponding MDB entry
  • option: --allocation
  • default: taken from the MDB
• Queue
  • The queue for the simulation, overriding the corresponding MDB entry
  • option: --queue
  • default: taken from the MDB
• Processors per node
  • The number of processors per node requested from the queueing system
  • option: --ppn
  • default: all processors on a node
• Used processors per node
  • The number of processors per node that should actually be used, allowing under-using nodes even if the queueing system does not allow it. (The remaining processors will idle and will remain unused.)
  • option: --ppn-used
  • default: all processors on a node

Running a Simulation

The Simulation Factory can execute a simulation directly, bypassing the queuing system. Running a simulation directly uses the same options, but ignores wall time limit etc. You use the run command for this:

simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

Additional Options: Running

See #Aditional Options: Submission

Other Simulation Commands

To launch an interactive session on a compute node, use the command

simfactory/bin/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise similar to the submit command.

To stop a simulation:

simfactory/bin/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/bin/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/bin/sim show-output <simulationname> [--restart-id=<restartid>]

What's Produced

When a simulation is submitted for the first time, all necessary information from the Cactus build configuration is brought into a specific simulation folder created underneath the basedir directory. Contained inside this folder, which has the same name as the specified simulation, are the executable, run script, submit script, a SIMFACTORY folder, a log file, and the output directories for each individual restart.

Simulations are self-contained, and once created do not rely on outside information. For example, recompiling the executable or changing the parameter file that were used to submit a simulation will not influence the simulation, since the simulation contains copies of both. This ensures that simulations can continue to run unperturbed even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r--  1 mwt lsuusers    0 Sep 30 13:30 LOG
drwxr-xr-x  3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x  4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x  4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x  3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x  4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x  7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script files for submission and execution, and a properties.ini file that is used by the Simulation Factory to store information about the simulation.

Each time a simulation is either run or submitted, a restart directory is created underneath the simulation directory. This restart folder has a name of the format "output-####", starting with "output-0000". Contained inside the restart folder are several internal files, the output written to stdout and stderr from the simulation, and the simulation output itself. The simulation output is typically stored inside a directory named after the basename of the parameter file. An example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r--  1 mwt lsuusers      0 Sep 17 09:06 LOG
-rw-r--r--  1 mwt lsuusers      9 Sep 17 09:06 mpd_nodefile
-rw-r--r--  1 mwt lsuusers     32 Sep 17 09:06 mpi_nodefile
-rw-r--r--  1 mwt lsuusers     33 Sep 17 09:06 NODELIST
drwxr-xr-x  3 mwt lsuusers  20480 Sep 17 16:12 qc0-mclachlan
-rw-------  1 mwt lsuusers   2520 Sep 17 21:06 qc0-mclachlan.err
-rw-------  1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r--  1 mwt lsuusers  13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx  1 mwt lsuusers     23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x  2 mwt lsuusers   4096 Sep 17 09:06 SIMFACTORY

Script Locations

When a simulation is created, it copies the submit script and the run script from the build configuration into the folder "basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/" folder, the run and submit scripts into the "run/" folder, the Cactus options list into the "cfg/" folder, and the parfile into the "par/" folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x  2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x  2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x  2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x  2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r--  1 mwt lsuusers  740 Sep 17 09:06 properties.ini
drwxr-xr-x  2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r--  1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x  1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r--  1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r--  1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r--  1 mwt lsuusers  410 Sep 17 09:06 SubmitScript

Other Advanced Features

Archiving

Preliminary support for archiving of simulations has been added to the Simulation Factory using the PetaShare data storage system on LONI. This archiving system can only be used from machines that have access to PetaShare. The currently supported machines that have access to Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare [http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz Pcommands], [http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php (manual)] must be downloaded and installed onto your machine. Once that is done, add a new section in etc/defs.local.ini for the machine from which you wish to access PetaShare, and add the necessary MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype      = petashare
archiveuser      = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath  = /tempZone/home/numrel/mwt/simulations

archivebasepath is where the simulations will be stored on PetaShare.

Once this setup has been completed, you can archive an entire simulation (including all restarts) using the command

simfactory/bin/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the command

simfactory/bin/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/bin/sim list-archived-simulations

To retreive an archived simulation, first use the list-archived-simulations command to retrieve the unique identifier for the simulation, and then issue the following command to retrieve a simulation. It will place the simulation in the current directory.

simfactory/bin/sim get-archived-simulation <archiveid>