Einstein Toolkit Documentation - User contributions [en]

Simulation Factory Interactive Debugging

2011-03-11T21:30:31Z

Mthomas:

The Simulation Factory provides a simple facility to interactively debug Cactus/Einstein Toolkit simulation sessions in parallel.
Currently, only the Totalview debugger is supported.

== Supported Machines ==

Interactive parallel debugging using Totalview is currently supported on the following machines:

<ul>
<li>queenbee (LONI/TeraGrid)
<li>eric (LONI)
<li>ranger (TeraGrid)
<li>abe (TeraGrid)
<li> lincoln (TeraGrid)
<li> kraken (TeraGrid)
</ul>

== Getting Started ==

Before debugging, a Cactus/Einstein Toolkit binary must be compiled, and then a Simulation Factory simulation must be created. Instructions to do these steps can be found here:

[https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial Simulation Factory Advanced Tutorial]

A second requirement is the existence of an Xwindows environment such as X11 or X-Ming.

== Interactive Sessions ==

To begin debugging, first launch an interactive session using The Simulation Factory. This can be done with the following command:

bin/sim interactive --walltime=01:00:00 --procs=8

walltime and procs can be changed to suit the amount of simulation time and processors needed.

== Interactive Debugging ==

Once the queuing system grants an interactive session, it will present a shell on a compute node. From here, it is
necessary to change the current working directory back to the simulation factory directory. Once located in The Simulation
Factory directory, an interactive debugging session can be launched by typing the following command:

bin/sim run-debug <simulationname>

For instance, if your simulation is called "test_sim", it would be:

bin/sim run-debug test_sim

Once the interactive debugging session launches. It is necessary to hit the "Go" button inside Totalview to begin execution of the Cactus/Einstein Toolkit simulation.

Simulation Factory Interactive Debugging

2011-03-11T21:30:13Z

Mthomas:

The Simulation Factory provides a simple facility to interactively debug Cactus/Einstein Toolkit simulation sessions in parallel.
Currently, only the Totalview debugger is supported.

== Supported Machines ==

Interactive parallel debugging using Totalview is currently supported on the following machines:

<ul>
<li>queenbee (LONI/TeraGrid)
<li>eric (LONI)
<li>ranger (TeraGrid)
<li>abe (TeraGrid)
<li> lincoln (TeraGrid)
<li> kraken (TeraGrid)
</ul>

== Getting Started ==

Before debugging, a Cactus/Einstein Toolkit binary must be compiled, and then a Simulation Factory simulation must be created. Instructions to do these steps can be found here:

[https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial Simulation_Factory_Advanced_Tutorial]

A second requirement is the existence of an Xwindows environment such as X11 or X-Ming.

== Interactive Sessions ==

To begin debugging, first launch an interactive session using The Simulation Factory. This can be done with the following command:

bin/sim interactive --walltime=01:00:00 --procs=8

walltime and procs can be changed to suit the amount of simulation time and processors needed.

== Interactive Debugging ==

Once the queuing system grants an interactive session, it will present a shell on a compute node. From here, it is
necessary to change the current working directory back to the simulation factory directory. Once located in The Simulation
Factory directory, an interactive debugging session can be launched by typing the following command:

bin/sim run-debug <simulationname>

For instance, if your simulation is called "test_sim", it would be:

bin/sim run-debug test_sim

Once the interactive debugging session launches. It is necessary to hit the "Go" button inside Totalview to begin execution of the Cactus/Einstein Toolkit simulation.

Simulation Factory Advanced Tutorial

2011-03-11T21:10:07Z

Mthomas:

The Simulation Factory simplifies many aspects of running Cactus-based
simulations. It provides a central facility for managing authoritative
source tree versions, providing convenient access to remote HPC
systems, building Cactus source configurations, and managing
simulations all the way from submission to archiving their output.

== Getting Started ==

To begin using The Simulation Factory, it needs to be checked out from
'''svn'''. The Simulation Factory is typically placed into a
'''simfactory''' folder inside a Cactus source tree. This can be
accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location
to be used with multiple Cactus source trees. This approach will be
described later.

== Initial Setup ==

Once the Simulation Factory has been checked out from svn, the next
step is to configure it, telling it e.g. about your user name. The
Simulation Factory comes with a convenient method to generate
some simple defaults to get started. To begin, issue the command

simfactory/bin/sim setup

Setup will prompt for a username, an email address, and an allocation. It will allow you
to enter additional configuration from other machines, but this is advanced, and can be
safely ignored. Setup will also generate a machine database entry for your local machine
based upon the generic machine database entry.

=== Additional Configurations ===

The Simulation Factory contains a database known as the Machine
Database. This collection of information describes all the aspects
that are unique about each individual HPC system, so that the
Simulation Factory can provide a common interface for all systems that
hides these differences.

The Machine Database consists of different sections, one for each
machine. The section name is given in square brackets, e.g.
'''[queenbee]'''. There is a special section '''[default]''' that
provides default values for those properties that are not explicitly
set in the machine-specific entries.

The Machine Database is an authoritative collection of information,
and is generally not meant to contain modification that are only
relevant for individual people. These local modifications are instead
maintained in the file '''simfactory/etc/defs.local.ini''' described
above, where one can add, change, or overwrite properties of Machine
Database entries. For instance, if an alternative username,
allocation, and sourcebasedir is needed for the machine
'''queenbee''', you would add the following section there:

[queenbee]
user = QUEENBEE_USERNAME
allocation = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries.
The most useful is probably '''@USER@''', which expands to the
'''user''' property of the Machine Database entry.

For example, if you are using the same user name on many systems, but
have a different user name on some systems, then you would set the
common user name in the '''[default]''' section, and override this for
those machines where your user name differs. The example
'''simfactory/etc/defs.local.ini.complex''' has examples for this.

Most of the macros available in the Simulation Factory are described
in the section [[#Macros]] below.

The command

simfactory/bin/sim list-machines

outputs a list of all preconfigured machines that the Simulation
Factory knows about.

=== Extended Local Workstation Configuration ===

Once determining the hostname of your machine through the command:

hostname

It is then possible to edit the machine simfactory/mdb/machines/<hostname>.ini, which was created via the command:

sim setup

Then edit '''simfactory/mdb/machines/<hostname>.ini'''. From here, all values can be customized.

== Accessing Remote Systems ==

The Simulation Factory simplifies access to remote systems, both for
transferring files and logging in. You can synchronise (replicate) an
authoritative version of your Cactus source tree to remote systems,
obtain an interactive shell, or execute commands.

=== Information Commands ===

The following commands can be used to discover information about a
machine, or list all known machines.

List all known machines:

simfactory/bin/sim list-machines

List details about a single machine:

simfactory/bin/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/bin/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/bin/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is
currently being executed:

simfactory/bin/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed
into a central location on each machine, but are instead built
on-demand by every user for a certain thorn list. (One of the
advantages is that people can thus easily add their own thorns.) To
help with this approach, the Simulation Factory provides a facility to
synchronize a Cactus user's local, authoritative source tree to remote
HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like
mechanisms such as gsi-ssh. Currently, you must still manage all ssh
keys and passwords manually. (We highly recommend to use ssh keychain
and ssh agents to avoid having to enter passwords multiple times.)

==== Configuration ====

Before syncing a source tree to a remote system, a small amount of
configuration must be performed. It is necessary to either verify that
the defaults are correct, or to define the correct values for the
following keys for the remote system in the Machine Database:

<ul>
<li> '''sourcebasedir'''
<ul>
<li>The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li>The root directory in which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li>The user name on the remote system
</ul>
</ul>

You can output the currently configured values by issuing the command

simfactory/bin/sim print-mdb <machine>

If you need to change these values, then edit (on the local system)
the file '''simfactory/etc/defs.local.ini''' and add a section for the
remote machine. This entry will augment the existing Machine Database
entry and updating/replace the corresponding values. An example for
the machine '''queenbee''' can be see in the [[#Additional
Configuration]] section.

To see the list of files and directories that are synchronized, look
at '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''sync-sources'''
<ul>
<li>The list of files and directories that will be transferred when
the option '''--sync-sourcetree''' is enabled (on by default)
</ul>
<li> '''sync-parfiles'''
<ul>
<li>The list of files and directories that will be copied when the
option '''--sync-parfiles''' is enabled (also on by default). This
list of files typically includes just parameter files.
</ul>
<li> '''sync-excludes'''
<ul>
<li>A list of files and directories that will be expressly excluded
from syncing, such as e.g. '''CVS''' or '''.svn''' directories.
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two options, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li>Synchronise the complete source tree, as specified in the
aforementioned '''rsync-sources''' configuration entry. This takes a
few seconds or minutes, depending on the connection.
</ul>
<li> '''sync-parfiles'''
<ul>
<li>Synchronise parameter files, as specified by the aforementioned
'''rsync-parfiles''' configuration entry. This is typically faster
than synchronising the source tree.
</ul>
</ul>

Usually, you would issue the command:

simfactory/bin/sim sync <machine>

To synchronise only parfiles, you can negate the
'''--sync-sourcetree''' argument with the following command

simfactory/bin/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from
another remote machine, then use

simfactory/bin/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine
'''<frommachine>'''.

=== Remote Login ===

The Simulation Factory provides the ability to log in to a remote
system. This is initiated with the command

simfactory/bin/sim login <machine>

This will automatically cd into the Cactus directory on the remote
system.

=== Local/Remote Command Execution ===

To execute a command (locally) via the Simulation Factory, use the
command

simfactory/bin/sim execute <command>

The command will be executed in the Cactus directory on the remote
system.

If the command is complex, and requires arguments, the command must be
quoted. For example

simfactory/bin/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/bin/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/bin/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

== Build Cactus Configurations ==

The Simulation Factory provides a central facility for configuring and
building Cactus source trees. When a Cactus source tree is compiled,
the Simulation Factory creates a '''configuration''' for the compiled
executable, storing with it related information such as the Cactus
options list, and the scripts necessary to submit and run jobs in a
queuing system. This configuration is thus a self-contained entity
containing everything that is necessary to perform Cactus simulations.

=== Information Commands ===

To list all existing Cactus configurations, use the following command

simfactory/bin/sim list-configurations

=== Building a Configuration ===

To build a configuration, four pieces of information are required:

<ul>
<li>Thorn list
<ul>
<li>This defines which thorns are to be included into the configuration.
<li>Default: '''thornlist''' parameter of the Machine Database entry
<li>Override: '''--thornlist=<thornlist>'''
<li>The default option list is probably not useful in many cases.
</ul>
<li>Option List
<ul>
<li>This specifies the compiler and build options that need to be used
to build Cactus on a particular system.
<li>Default: '''optionlist''' parameter of the Machine Database entry
<li>Override: '''--optionlist=<optionlist>'''
<li>The Simulation Factory is supposed to contain good, working
default option lists for all supported systems. In fact, this is one
of the main strengths of the Simulation Factory. You should normally
not need to override the default.
</ul>
<li>Submission Script
<ul>
<li>This specifies how to submit a job to the queueing system on a
particular system.
<li>Default: '''submitscript''' parameter of the Machine Database entry
<li>Override: '''--submitscript=<submitscript>'''
<li>Similar to the option list, the Simulation Factory is supposed to
contain good, working default submission scripts for all supported
systems.
</ul>
<li>Run Script
<ul>
<li>This specifies to how execute an MPI process on a particular
system; it is closely connected to the submission script.
<li>Default: '''runscript''' parameter of the Machine Database entry
<li>Override: '''--runscript=<runscript>'''
<li>Same as with the the submission script, the Simulation Factory is
supposed to contain good, working default run scripts for all
supported systems.
</ul>
</ul>

To build a configuration with a specific thornlist, issue the
following command:

simfactory/bin/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to
'sim'. (We recommend this.) You can in addition specify any of the
options
<ul>
<li>--debug
<li>--optimise (default)
<li>--profile
</ul>
These options create configurations for debugging, optimisation (this
is the default), or profiling enabled.
If any of these options is specified, then the configuration name will
be modified correspondingly, e.g. to 'sim-debug'.

==== Additional Options ====

<ul>

<li>'''--reconfig'''
<ul>
<li>Reconfigure before building, i.e. re-examine the configuration
options and re-run the CST stage. This happens automatically when
the option list changes.
</ul>

<li>'''--clean'''
<ul>
<li>Clean the configuration (remove all object files etc.) before
building.
</ul>

</ul>

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

=== Script Locations ===

The Simulation Factory provides default scripts for all its
preconfigured machines. These scripts can be found in the following
locations

<ul>

<li>'''Option Lists'''
<ul>
<li>MDB Key: optionlist
<li>Location: simfactory/etc/optionlists
</ul>

<li>'''Submit Scripts'''
<ul>
<li>MDB Key: submitscript
<li>Location: simfactory/etc/submitscripts
</ul>

<li>'''Run Scripts'''
<ul>
<li>MDB Key: runscript
<li>Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list Queen Bee uses by
default, issue the command

simfactory/bin/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for
submitting, running, and managing simulations. This is accomplished
through two main commands '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations on a particular machine can be seen with
the following command

simfactory/bin/sim list-simulations

If a more detailed look at each simulation is required, the verbose
option can be specified

simfactory/bin/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a
simulation to the host queuing system. They are

<ul>

<li>'''Configuration'''
<ul>
<li>The Cactus configuration to run
<li>'''option''': --configuration
<li>'''default''': "sim"
</ul>

<li>'''Parfile'''
<ul>
<li>The Cactus parameter file to use
<li>'''option''': --parfile
</ul>

<li>'''Walltime'''
<ul>
<li>The total amount of wall time required
<li>'''option''': --walltime
<li>'''default''': MDB Key '''maxwalltime'''
</ul>

<li>Processors
<ul>
<li>The total number of processors to use
<li>'''option''': --procs
<li>'''default''': 1
</ul>

</ul>

The option '''--configuration''' only needs to be specified the first
time you submit a simulation. Subsequent re-submissions of the same
simulation (for restarting from checkpoints) will always use the same
configuration that was specified the first time. Here is an example of
submitting a simulation named "static_tov" using the aforementioned
options:

simfactory/bin/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where
you do not need to specify the option names, but have to specify the
options in a certain order. If you don't specify a simulation name
using the shorthand syntax, a simulation name will be derived from
from the parameter file name.

simfactory/bin/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/bin/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li>Number of OpenMP Threads
<ul>
<li>The number of OpenMP threads per MPI process. (You specify the
total number of processors (cores), and the number of OpenMP
threads; the number of MPI processes is then calculated
automatically.)
<li>option: --num-threads
<li>default: 1 (as if OpenMP was not used)
</ul>

<li>Allocation
<ul>
<li>The allocation for the simulation, overriding the corresponding
MDB entry
<li>option: --allocation
<li>default: taken from the MDB
</ul>

<li>Queue
<ul>
<li>The queue for the simulation, overriding the corresponding MDB
entry
<li>option: --queue
<li>default: taken from the MDB
</ul>

<li>Processors per node
<ul>
<li>The number of processors per node requested from the queueing system
<li>option: --ppn
<li>default: all processors on a node
</ul>

<li>Used processors per node
<ul>
<li>The number of processors per node that should actually be used,
allowing under-using nodes even if the queueing system does not
allow it. (The remaining processors will idle and will remain
unused.)
<li>option: --ppn-used
<li>default: all processors on a node
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing
the queuing system. Running a simulation directly uses the same
options, but ignores wall time limit etc. You use the '''run'''
command for this:

simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the command

simfactory/bin/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise
similar to the submit command.

To stop a simulation:

simfactory/bin/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/bin/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/bin/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is submitted for the first time, all necessary
information from the Cactus build configuration is brought into a
specific simulation folder created underneath the '''basedir'''
directory. Contained inside this folder, which has the same name as
the specified simulation, are the executable, run script, submit
script, a SIMFACTORY folder, a log file, and the output directories
for each individual restart.

Simulations are self-contained, and once created do not rely on
outside information. For example, recompiling the executable or
changing the parameter file that were used to submit a simulation will
not influence the simulation, since the simulation contains copies of
both. This ensures that simulations can continue to run unperturbed
even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with
several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script
files for submission and execution, and a properties.ini file that is
used by the Simulation Factory to store information about the
simulation.

Each time a simulation is either run or submitted, a restart directory
is created underneath the simulation directory. This restart folder
has a name of the format "output-####", starting with "output-0000".
Contained inside the restart folder are several internal files, the
output written to stdout and stderr from the simulation, and the
simulation output itself. The simulation output is typically stored
inside a directory named after the basename of the parameter file. An
example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run
script from the build configuration into the folder
"basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/"
folder, the run and submit scripts into the "run/" folder, the Cactus
options list into the "cfg/" folder, and the parfile into the "par/"
folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to the
Simulation Factory using the PetaShare data storage system on LONI.
This archiving system can only be used from machines that have access
to PetaShare. The currently supported machines that have access to
Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare
[http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz
Pcommands],
[http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php
(manual)] must be downloaded and installed onto your machine. Once
that is done, add a new section in '''etc/defs.local.ini''' for the
machine from which you wish to access PetaShare, and add the necessary
MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

'''archivebasepath''' is where the simulations will be stored on
PetaShare.

Once this setup has been completed, you can archive an entire
simulation (including all restarts) using the command

simfactory/bin/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the
command

simfactory/bin/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/bin/sim list-archived-simulations

To retreive an archived simulation, first use the
'''list-archived-simulations''' command to retrieve the unique
identifier for the simulation, and then issue the following command to
retrieve a simulation. It will place the simulation in the current
directory.

simfactory/bin/sim get-archived-simulation <archiveid>

Simulation Factory Advanced Tutorial

2011-03-11T20:36:04Z

Mthomas:

The Simulation Factory simplifies many aspects of running Cactus-based
simulations. It provides a central facility for managing authoritative
source tree versions, providing convenient access to remote HPC
systems, building Cactus source configurations, and managing
simulations all the way from submission to archiving their output.

== Getting Started ==

To begin using The Simulation Factory, it needs to be checked out from
'''svn'''. The Simulation Factory is typically placed into a
'''simfactory''' folder inside a Cactus source tree. This can be
accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location
to be used with multiple Cactus source trees. This approach will be
described later.

== Initial Setup ==

Once the Simulation Factory has been checked out from svn, the next
step is to configure it, telling it e.g. about your user name. The
Simulation Factory comes with a convenient method to generate
some simple defaults to get started. To begin, issue the command

simfactory/bin/sim setup

Setup will prompt for a username, an email address, and an allocation. It will allow you
to enter additional configuration from other machines, but this is advanced, and can be
safely ignored. Setup will also generate a machine database entry for your local machine
based upon the generic machine database entry.

=== Additional Configurations ===

The Simulation Factory contains a database known as the Machine
Database. This collection of information describes all the aspects
that are unique about each individual HPC system, so that the
Simulation Factory can provide a common interface for all systems that
hides these differences.

The Machine Database consists of different sections, one for each
machine. The section name is given in square brackets, e.g.
'''[queenbee]'''. There is a special section '''[default]''' that
provides default values for those properties that are not explicitly
set in the machine-specific entries.

The Machine Database is an authoritative collection of information,
and is generally not meant to contain modification that are only
relevant for individual people. These local modifications are instead
maintained in the file '''simfactory/etc/defs.local.ini''' described
above, where one can add, change, or overwrite properties of Machine
Database entries. For instance, if an alternative username,
allocation, and sourcebasedir is needed for the machine
'''queenbee''', you would add the following section there:

[queenbee]
user = QUEENBEE_USERNAME
allocation = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries.
The most useful is probably '''@USER@''', which expands to the
'''user''' property of the Machine Database entry.

For example, if you are using the same user name on many systems, but
have a different user name on some systems, then you would set the
common user name in the '''[default]''' section, and override this for
those machines where your user name differs. The example
'''simfactory/etc/defs.local.ini.complex''' has examples for this.

Most of the macros available in the Simulation Factory are described
in the section [[#Macros]] below.

The command

simfactory/bin/sim list-machines

outputs a list of all preconfigured machines that the Simulation
Factory knows about.

=== Local Workstation Configuration ===

The Simulation Factory can only be used on a machine known to the
Simulation Factory. This means that you may have to add an entry for
your workstation or notebook.

The first step is to determine the hostname of the local machine. It
is through this hostname that the Simulation Factory matches a Machine
Database entry to the machine on which it executes. The hostname can
be determined using the following command:

hostname

Once you know the hostname, issue the following command to create a
new Machine Database entry, starting from a generic example:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Then edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li>The section header for this machine database entry must be unique
among all Machine Database entries, and must match the '''nickname'''
property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root'''
path under which all your Cactus source trees reside. (This is not
the Cactus directory itself, but the directory that contains the
Cactus directory.)
<li> '''basedir = /home/@USER@/simulations''' with the desired folder
that will contain all simulation output. (This is explained in more
detail below in the section [[#Managing Simulations]].)
</ul>
You can ignore the properties '''user''', '''email''', and
'''allocation''', as the values from the '''[default]''' section of
'''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory simplifies access to remote systems, both for
transferring files and logging in. You can synchronise (replicate) an
authoritative version of your Cactus source tree to remote systems,
obtain an interactive shell, or execute commands.

=== Information Commands ===

The following commands can be used to discover information about a
machine, or list all known machines.

List all known machines:

simfactory/bin/sim list-machines

List details about a single machine:

simfactory/bin/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/bin/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/bin/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is
currently being executed:

simfactory/bin/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed
into a central location on each machine, but are instead built
on-demand by every user for a certain thorn list. (One of the
advantages is that people can thus easily add their own thorns.) To
help with this approach, the Simulation Factory provides a facility to
synchronize a Cactus user's local, authoritative source tree to remote
HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like
mechanisms such as gsi-ssh. Currently, you must still manage all ssh
keys and passwords manually. (We highly recommend to use ssh keychain
and ssh agents to avoid having to enter passwords multiple times.)

==== Configuration ====

Before syncing a source tree to a remote system, a small amount of
configuration must be performed. It is necessary to either verify that
the defaults are correct, or to define the correct values for the
following keys for the remote system in the Machine Database:

<ul>
<li> '''sourcebasedir'''
<ul>
<li>The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li>The root directory in which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li>The user name on the remote system
</ul>
</ul>

You can output the currently configured values by issuing the command

simfactory/bin/sim print-mdb <machine>

If you need to change these values, then edit (on the local system)
the file '''simfactory/etc/defs.local.ini''' and add a section for the
remote machine. This entry will augment the existing Machine Database
entry and updating/replace the corresponding values. An example for
the machine '''queenbee''' can be see in the [[#Additional
Configuration]] section.

To see the list of files and directories that are synchronized, look
at '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''sync-sources'''
<ul>
<li>The list of files and directories that will be transferred when
the option '''--sync-sourcetree''' is enabled (on by default)
</ul>
<li> '''sync-parfiles'''
<ul>
<li>The list of files and directories that will be copied when the
option '''--sync-parfiles''' is enabled (also on by default). This
list of files typically includes just parameter files.
</ul>
<li> '''sync-excludes'''
<ul>
<li>A list of files and directories that will be expressly excluded
from syncing, such as e.g. '''CVS''' or '''.svn''' directories.
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two options, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li>Synchronise the complete source tree, as specified in the
aforementioned '''rsync-sources''' configuration entry. This takes a
few seconds or minutes, depending on the connection.
</ul>
<li> '''sync-parfiles'''
<ul>
<li>Synchronise parameter files, as specified by the aforementioned
'''rsync-parfiles''' configuration entry. This is typically faster
than synchronising the source tree.
</ul>
</ul>

Usually, you would issue the command:

simfactory/bin/sim sync <machine>

To synchronise only parfiles, you can negate the
'''--sync-sourcetree''' argument with the following command

simfactory/bin/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from
another remote machine, then use

simfactory/bin/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine
'''<frommachine>'''.

=== Remote Login ===

The Simulation Factory provides the ability to log in to a remote
system. This is initiated with the command

simfactory/bin/sim login <machine>

This will automatically cd into the Cactus directory on the remote
system.

=== Local/Remote Command Execution ===

To execute a command (locally) via the Simulation Factory, use the
command

simfactory/bin/sim execute <command>

The command will be executed in the Cactus directory on the remote
system.

If the command is complex, and requires arguments, the command must be
quoted. For example

simfactory/bin/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/bin/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/bin/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

== Build Cactus Configurations ==

The Simulation Factory provides a central facility for configuring and
building Cactus source trees. When a Cactus source tree is compiled,
the Simulation Factory creates a '''configuration''' for the compiled
executable, storing with it related information such as the Cactus
options list, and the scripts necessary to submit and run jobs in a
queuing system. This configuration is thus a self-contained entity
containing everything that is necessary to perform Cactus simulations.

=== Information Commands ===

To list all existing Cactus configurations, use the following command

simfactory/bin/sim list-configurations

=== Building a Configuration ===

To build a configuration, four pieces of information are required:

<ul>
<li>Thorn list
<ul>
<li>This defines which thorns are to be included into the configuration.
<li>Default: '''thornlist''' parameter of the Machine Database entry
<li>Override: '''--thornlist=<thornlist>'''
<li>The default option list is probably not useful in many cases.
</ul>
<li>Option List
<ul>
<li>This specifies the compiler and build options that need to be used
to build Cactus on a particular system.
<li>Default: '''optionlist''' parameter of the Machine Database entry
<li>Override: '''--optionlist=<optionlist>'''
<li>The Simulation Factory is supposed to contain good, working
default option lists for all supported systems. In fact, this is one
of the main strengths of the Simulation Factory. You should normally
not need to override the default.
</ul>
<li>Submission Script
<ul>
<li>This specifies how to submit a job to the queueing system on a
particular system.
<li>Default: '''submitscript''' parameter of the Machine Database entry
<li>Override: '''--submitscript=<submitscript>'''
<li>Similar to the option list, the Simulation Factory is supposed to
contain good, working default submission scripts for all supported
systems.
</ul>
<li>Run Script
<ul>
<li>This specifies to how execute an MPI process on a particular
system; it is closely connected to the submission script.
<li>Default: '''runscript''' parameter of the Machine Database entry
<li>Override: '''--runscript=<runscript>'''
<li>Same as with the the submission script, the Simulation Factory is
supposed to contain good, working default run scripts for all
supported systems.
</ul>
</ul>

To build a configuration with a specific thornlist, issue the
following command:

simfactory/bin/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to
'sim'. (We recommend this.) You can in addition specify any of the
options
<ul>
<li>--debug
<li>--optimise (default)
<li>--profile
</ul>
These options create configurations for debugging, optimisation (this
is the default), or profiling enabled.
If any of these options is specified, then the configuration name will
be modified correspondingly, e.g. to 'sim-debug'.

==== Additional Options ====

<ul>

<li>'''--reconfig'''
<ul>
<li>Reconfigure before building, i.e. re-examine the configuration
options and re-run the CST stage. This happens automatically when
the option list changes.
</ul>

<li>'''--clean'''
<ul>
<li>Clean the configuration (remove all object files etc.) before
building.
</ul>

</ul>

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

=== Script Locations ===

The Simulation Factory provides default scripts for all its
preconfigured machines. These scripts can be found in the following
locations

<ul>

<li>'''Option Lists'''
<ul>
<li>MDB Key: optionlist
<li>Location: simfactory/etc/optionlists
</ul>

<li>'''Submit Scripts'''
<ul>
<li>MDB Key: submitscript
<li>Location: simfactory/etc/submitscripts
</ul>

<li>'''Run Scripts'''
<ul>
<li>MDB Key: runscript
<li>Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list Queen Bee uses by
default, issue the command

simfactory/bin/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for
submitting, running, and managing simulations. This is accomplished
through two main commands '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations on a particular machine can be seen with
the following command

simfactory/bin/sim list-simulations

If a more detailed look at each simulation is required, the verbose
option can be specified

simfactory/bin/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a
simulation to the host queuing system. They are

<ul>

<li>'''Configuration'''
<ul>
<li>The Cactus configuration to run
<li>'''option''': --configuration
<li>'''default''': "sim"
</ul>

<li>'''Parfile'''
<ul>
<li>The Cactus parameter file to use
<li>'''option''': --parfile
</ul>

<li>'''Walltime'''
<ul>
<li>The total amount of wall time required
<li>'''option''': --walltime
<li>'''default''': MDB Key '''maxwalltime'''
</ul>

<li>Processors
<ul>
<li>The total number of processors to use
<li>'''option''': --procs
<li>'''default''': 1
</ul>

</ul>

The option '''--configuration''' only needs to be specified the first
time you submit a simulation. Subsequent re-submissions of the same
simulation (for restarting from checkpoints) will always use the same
configuration that was specified the first time. Here is an example of
submitting a simulation named "static_tov" using the aforementioned
options:

simfactory/bin/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where
you do not need to specify the option names, but have to specify the
options in a certain order. If you don't specify a simulation name
using the shorthand syntax, a simulation name will be derived from
from the parameter file name.

simfactory/bin/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/bin/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li>Number of OpenMP Threads
<ul>
<li>The number of OpenMP threads per MPI process. (You specify the
total number of processors (cores), and the number of OpenMP
threads; the number of MPI processes is then calculated
automatically.)
<li>option: --num-threads
<li>default: 1 (as if OpenMP was not used)
</ul>

<li>Allocation
<ul>
<li>The allocation for the simulation, overriding the corresponding
MDB entry
<li>option: --allocation
<li>default: taken from the MDB
</ul>

<li>Queue
<ul>
<li>The queue for the simulation, overriding the corresponding MDB
entry
<li>option: --queue
<li>default: taken from the MDB
</ul>

<li>Processors per node
<ul>
<li>The number of processors per node requested from the queueing system
<li>option: --ppn
<li>default: all processors on a node
</ul>

<li>Used processors per node
<ul>
<li>The number of processors per node that should actually be used,
allowing under-using nodes even if the queueing system does not
allow it. (The remaining processors will idle and will remain
unused.)
<li>option: --ppn-used
<li>default: all processors on a node
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing
the queuing system. Running a simulation directly uses the same
options, but ignores wall time limit etc. You use the '''run'''
command for this:

simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the command

simfactory/bin/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise
similar to the submit command.

To stop a simulation:

simfactory/bin/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/bin/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/bin/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is submitted for the first time, all necessary
information from the Cactus build configuration is brought into a
specific simulation folder created underneath the '''basedir'''
directory. Contained inside this folder, which has the same name as
the specified simulation, are the executable, run script, submit
script, a SIMFACTORY folder, a log file, and the output directories
for each individual restart.

Simulations are self-contained, and once created do not rely on
outside information. For example, recompiling the executable or
changing the parameter file that were used to submit a simulation will
not influence the simulation, since the simulation contains copies of
both. This ensures that simulations can continue to run unperturbed
even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with
several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script
files for submission and execution, and a properties.ini file that is
used by the Simulation Factory to store information about the
simulation.

Each time a simulation is either run or submitted, a restart directory
is created underneath the simulation directory. This restart folder
has a name of the format "output-####", starting with "output-0000".
Contained inside the restart folder are several internal files, the
output written to stdout and stderr from the simulation, and the
simulation output itself. The simulation output is typically stored
inside a directory named after the basename of the parameter file. An
example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run
script from the build configuration into the folder
"basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/"
folder, the run and submit scripts into the "run/" folder, the Cactus
options list into the "cfg/" folder, and the parfile into the "par/"
folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to the
Simulation Factory using the PetaShare data storage system on LONI.
This archiving system can only be used from machines that have access
to PetaShare. The currently supported machines that have access to
Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare
[http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz
Pcommands],
[http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php
(manual)] must be downloaded and installed onto your machine. Once
that is done, add a new section in '''etc/defs.local.ini''' for the
machine from which you wish to access PetaShare, and add the necessary
MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

'''archivebasepath''' is where the simulations will be stored on
PetaShare.

Once this setup has been completed, you can archive an entire
simulation (including all restarts) using the command

simfactory/bin/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the
command

simfactory/bin/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/bin/sim list-archived-simulations

To retreive an archived simulation, first use the
'''list-archived-simulations''' command to retrieve the unique
identifier for the simulation, and then issue the following command to
retrieve a simulation. It will place the simulation in the current
directory.

simfactory/bin/sim get-archived-simulation <archiveid>

Simulation Factory Advanced Tutorial

2011-03-11T20:31:32Z

Mthomas:

The Simulation Factory simplifies many aspects of running Cactus-based
simulations. It provides a central facility for managing authoritative
source tree versions, providing convenient access to remote HPC
systems, building Cactus source configurations, and managing
simulations all the way from submission to archiving their output.

== Getting Started ==

To begin using The Simulation Factory, it needs to be checked out from
'''svn'''. The Simulation Factory is typically placed into a
'''simfactory''' folder inside a Cactus source tree. This can be
accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location
to be used with multiple Cactus source trees. This approach will be
described later.

== Initial Setup ==

Once the Simulation Factory has been checked out from svn, the next
step is to configure it, telling it e.g. about your user name. The
Simulation Factory comes with example configuration files that you can
copy and modify. Assuming The Simulation Factory has been checked out
into the '''simfactory''' folder, this initial configuration can be
accomplished with the following commands:

cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configurations ===

The Simulation Factory contains a database known as the Machine
Database. This collection of information describes all the aspects
that are unique about each individual HPC system, so that the
Simulation Factory can provide a common interface for all systems that
hides these differences.

The Machine Database consists of different sections, one for each
machine. The section name is given in square brackets, e.g.
'''[queenbee]'''. There is a special section '''[default]''' that
provides default values for those properties that are not explicitly
set in the machine-specific entries.

The Machine Database is an authoritative collection of information,
and is generally not meant to contain modification that are only
relevant for individual people. These local modifications are instead
maintained in the file '''simfactory/etc/defs.local.ini''' described
above, where one can add, change, or overwrite properties of Machine
Database entries. For instance, if an alternative username,
allocation, and sourcebasedir is needed for the machine
'''queenbee''', you would add the following section there:

[queenbee]
user = QUEENBEE_USERNAME
allocation = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries.
The most useful is probably '''@USER@''', which expands to the
'''user''' property of the Machine Database entry.

For example, if you are using the same user name on many systems, but
have a different user name on some systems, then you would set the
common user name in the '''[default]''' section, and override this for
those machines where your user name differs. The example
'''simfactory/etc/defs.local.ini.complex''' has examples for this.

Most of the macros available in the Simulation Factory are described
in the section [[#Macros]] below.

The command

simfactory/bin/sim list-machines

outputs a list of all preconfigured machines that the Simulation
Factory knows about.

=== Local Workstation Configuration ===

The Simulation Factory can only be used on a machine known to the
Simulation Factory. This means that you may have to add an entry for
your workstation or notebook.

The first step is to determine the hostname of the local machine. It
is through this hostname that the Simulation Factory matches a Machine
Database entry to the machine on which it executes. The hostname can
be determined using the following command:

hostname

Once you know the hostname, issue the following command to create a
new Machine Database entry, starting from a generic example:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Then edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li>The section header for this machine database entry must be unique
among all Machine Database entries, and must match the '''nickname'''
property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root'''
path under which all your Cactus source trees reside. (This is not
the Cactus directory itself, but the directory that contains the
Cactus directory.)
<li> '''basedir = /home/@USER@/simulations''' with the desired folder
that will contain all simulation output. (This is explained in more
detail below in the section [[#Managing Simulations]].)
</ul>
You can ignore the properties '''user''', '''email''', and
'''allocation''', as the values from the '''[default]''' section of
'''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory simplifies access to remote systems, both for
transferring files and logging in. You can synchronise (replicate) an
authoritative version of your Cactus source tree to remote systems,
obtain an interactive shell, or execute commands.

=== Information Commands ===

The following commands can be used to discover information about a
machine, or list all known machines.

List all known machines:

simfactory/bin/sim list-machines

List details about a single machine:

simfactory/bin/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/bin/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/bin/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is
currently being executed:

simfactory/bin/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed
into a central location on each machine, but are instead built
on-demand by every user for a certain thorn list. (One of the
advantages is that people can thus easily add their own thorns.) To
help with this approach, the Simulation Factory provides a facility to
synchronize a Cactus user's local, authoritative source tree to remote
HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like
mechanisms such as gsi-ssh. Currently, you must still manage all ssh
keys and passwords manually. (We highly recommend to use ssh keychain
and ssh agents to avoid having to enter passwords multiple times.)

==== Configuration ====

Before syncing a source tree to a remote system, a small amount of
configuration must be performed. It is necessary to either verify that
the defaults are correct, or to define the correct values for the
following keys for the remote system in the Machine Database:

<ul>
<li> '''sourcebasedir'''
<ul>
<li>The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li>The root directory in which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li>The user name on the remote system
</ul>
</ul>

You can output the currently configured values by issuing the command

simfactory/bin/sim print-mdb <machine>

If you need to change these values, then edit (on the local system)
the file '''simfactory/etc/defs.local.ini''' and add a section for the
remote machine. This entry will augment the existing Machine Database
entry and updating/replace the corresponding values. An example for
the machine '''queenbee''' can be see in the [[#Additional
Configuration]] section.

To see the list of files and directories that are synchronized, look
at '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''sync-sources'''
<ul>
<li>The list of files and directories that will be transferred when
the option '''--sync-sourcetree''' is enabled (on by default)
</ul>
<li> '''sync-parfiles'''
<ul>
<li>The list of files and directories that will be copied when the
option '''--sync-parfiles''' is enabled (also on by default). This
list of files typically includes just parameter files.
</ul>
<li> '''sync-excludes'''
<ul>
<li>A list of files and directories that will be expressly excluded
from syncing, such as e.g. '''CVS''' or '''.svn''' directories.
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two options, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li>Synchronise the complete source tree, as specified in the
aforementioned '''rsync-sources''' configuration entry. This takes a
few seconds or minutes, depending on the connection.
</ul>
<li> '''sync-parfiles'''
<ul>
<li>Synchronise parameter files, as specified by the aforementioned
'''rsync-parfiles''' configuration entry. This is typically faster
than synchronising the source tree.
</ul>
</ul>

Usually, you would issue the command:

simfactory/bin/sim sync <machine>

To synchronise only parfiles, you can negate the
'''--sync-sourcetree''' argument with the following command

simfactory/bin/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from
another remote machine, then use

simfactory/bin/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine
'''<frommachine>'''.

=== Remote Login ===

The Simulation Factory provides the ability to log in to a remote
system. This is initiated with the command

simfactory/bin/sim login <machine>

This will automatically cd into the Cactus directory on the remote
system.

=== Local/Remote Command Execution ===

To execute a command (locally) via the Simulation Factory, use the
command

simfactory/bin/sim execute <command>

The command will be executed in the Cactus directory on the remote
system.

If the command is complex, and requires arguments, the command must be
quoted. For example

simfactory/bin/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/bin/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/bin/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

== Build Cactus Configurations ==

The Simulation Factory provides a central facility for configuring and
building Cactus source trees. When a Cactus source tree is compiled,
the Simulation Factory creates a '''configuration''' for the compiled
executable, storing with it related information such as the Cactus
options list, and the scripts necessary to submit and run jobs in a
queuing system. This configuration is thus a self-contained entity
containing everything that is necessary to perform Cactus simulations.

=== Information Commands ===

To list all existing Cactus configurations, use the following command

simfactory/bin/sim list-configurations

=== Building a Configuration ===

To build a configuration, four pieces of information are required:

<ul>
<li>Thorn list
<ul>
<li>This defines which thorns are to be included into the configuration.
<li>Default: '''thornlist''' parameter of the Machine Database entry
<li>Override: '''--thornlist=<thornlist>'''
<li>The default option list is probably not useful in many cases.
</ul>
<li>Option List
<ul>
<li>This specifies the compiler and build options that need to be used
to build Cactus on a particular system.
<li>Default: '''optionlist''' parameter of the Machine Database entry
<li>Override: '''--optionlist=<optionlist>'''
<li>The Simulation Factory is supposed to contain good, working
default option lists for all supported systems. In fact, this is one
of the main strengths of the Simulation Factory. You should normally
not need to override the default.
</ul>
<li>Submission Script
<ul>
<li>This specifies how to submit a job to the queueing system on a
particular system.
<li>Default: '''submitscript''' parameter of the Machine Database entry
<li>Override: '''--submitscript=<submitscript>'''
<li>Similar to the option list, the Simulation Factory is supposed to
contain good, working default submission scripts for all supported
systems.
</ul>
<li>Run Script
<ul>
<li>This specifies to how execute an MPI process on a particular
system; it is closely connected to the submission script.
<li>Default: '''runscript''' parameter of the Machine Database entry
<li>Override: '''--runscript=<runscript>'''
<li>Same as with the the submission script, the Simulation Factory is
supposed to contain good, working default run scripts for all
supported systems.
</ul>
</ul>

To build a configuration with a specific thornlist, issue the
following command:

simfactory/bin/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to
'sim'. (We recommend this.) You can in addition specify any of the
options
<ul>
<li>--debug
<li>--optimise (default)
<li>--profile
</ul>
These options create configurations for debugging, optimisation (this
is the default), or profiling enabled.
If any of these options is specified, then the configuration name will
be modified correspondingly, e.g. to 'sim-debug'.

==== Additional Options ====

<ul>

<li>'''--reconfig'''
<ul>
<li>Reconfigure before building, i.e. re-examine the configuration
options and re-run the CST stage. This happens automatically when
the option list changes.
</ul>

<li>'''--clean'''
<ul>
<li>Clean the configuration (remove all object files etc.) before
building.
</ul>

</ul>

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

=== Script Locations ===

The Simulation Factory provides default scripts for all its
preconfigured machines. These scripts can be found in the following
locations

<ul>

<li>'''Option Lists'''
<ul>
<li>MDB Key: optionlist
<li>Location: simfactory/etc/optionlists
</ul>

<li>'''Submit Scripts'''
<ul>
<li>MDB Key: submitscript
<li>Location: simfactory/etc/submitscripts
</ul>

<li>'''Run Scripts'''
<ul>
<li>MDB Key: runscript
<li>Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list Queen Bee uses by
default, issue the command

simfactory/bin/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for
submitting, running, and managing simulations. This is accomplished
through two main commands '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations on a particular machine can be seen with
the following command

simfactory/bin/sim list-simulations

If a more detailed look at each simulation is required, the verbose
option can be specified

simfactory/bin/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a
simulation to the host queuing system. They are

<ul>

<li>'''Configuration'''
<ul>
<li>The Cactus configuration to run
<li>'''option''': --configuration
<li>'''default''': "sim"
</ul>

<li>'''Parfile'''
<ul>
<li>The Cactus parameter file to use
<li>'''option''': --parfile
</ul>

<li>'''Walltime'''
<ul>
<li>The total amount of wall time required
<li>'''option''': --walltime
<li>'''default''': MDB Key '''maxwalltime'''
</ul>

<li>Processors
<ul>
<li>The total number of processors to use
<li>'''option''': --procs
<li>'''default''': 1
</ul>

</ul>

The option '''--configuration''' only needs to be specified the first
time you submit a simulation. Subsequent re-submissions of the same
simulation (for restarting from checkpoints) will always use the same
configuration that was specified the first time. Here is an example of
submitting a simulation named "static_tov" using the aforementioned
options:

simfactory/bin/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where
you do not need to specify the option names, but have to specify the
options in a certain order. If you don't specify a simulation name
using the shorthand syntax, a simulation name will be derived from
from the parameter file name.

simfactory/bin/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/bin/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li>Number of OpenMP Threads
<ul>
<li>The number of OpenMP threads per MPI process. (You specify the
total number of processors (cores), and the number of OpenMP
threads; the number of MPI processes is then calculated
automatically.)
<li>option: --num-threads
<li>default: 1 (as if OpenMP was not used)
</ul>

<li>Allocation
<ul>
<li>The allocation for the simulation, overriding the corresponding
MDB entry
<li>option: --allocation
<li>default: taken from the MDB
</ul>

<li>Queue
<ul>
<li>The queue for the simulation, overriding the corresponding MDB
entry
<li>option: --queue
<li>default: taken from the MDB
</ul>

<li>Processors per node
<ul>
<li>The number of processors per node requested from the queueing system
<li>option: --ppn
<li>default: all processors on a node
</ul>

<li>Used processors per node
<ul>
<li>The number of processors per node that should actually be used,
allowing under-using nodes even if the queueing system does not
allow it. (The remaining processors will idle and will remain
unused.)
<li>option: --ppn-used
<li>default: all processors on a node
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing
the queuing system. Running a simulation directly uses the same
options, but ignores wall time limit etc. You use the '''run'''
command for this:

simfactory/bin/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the command

simfactory/bin/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise
similar to the submit command.

To stop a simulation:

simfactory/bin/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/bin/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/bin/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is submitted for the first time, all necessary
information from the Cactus build configuration is brought into a
specific simulation folder created underneath the '''basedir'''
directory. Contained inside this folder, which has the same name as
the specified simulation, are the executable, run script, submit
script, a SIMFACTORY folder, a log file, and the output directories
for each individual restart.

Simulations are self-contained, and once created do not rely on
outside information. For example, recompiling the executable or
changing the parameter file that were used to submit a simulation will
not influence the simulation, since the simulation contains copies of
both. This ensures that simulations can continue to run unperturbed
even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with
several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script
files for submission and execution, and a properties.ini file that is
used by the Simulation Factory to store information about the
simulation.

Each time a simulation is either run or submitted, a restart directory
is created underneath the simulation directory. This restart folder
has a name of the format "output-####", starting with "output-0000".
Contained inside the restart folder are several internal files, the
output written to stdout and stderr from the simulation, and the
simulation output itself. The simulation output is typically stored
inside a directory named after the basename of the parameter file. An
example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run
script from the build configuration into the folder
"basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/"
folder, the run and submit scripts into the "run/" folder, the Cactus
options list into the "cfg/" folder, and the parfile into the "par/"
folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to the
Simulation Factory using the PetaShare data storage system on LONI.
This archiving system can only be used from machines that have access
to PetaShare. The currently supported machines that have access to
Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare
[http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz
Pcommands],
[http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php
(manual)] must be downloaded and installed onto your machine. Once
that is done, add a new section in '''etc/defs.local.ini''' for the
machine from which you wish to access PetaShare, and add the necessary
MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

'''archivebasepath''' is where the simulations will be stored on
PetaShare.

Once this setup has been completed, you can archive an entire
simulation (including all restarts) using the command

simfactory/bin/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the
command

simfactory/bin/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/bin/sim list-archived-simulations

To retreive an archived simulation, first use the
'''list-archived-simulations''' command to retrieve the unique
identifier for the simulation, and then issue the following command to
retrieve a simulation. It will place the simulation in the current
directory.

simfactory/bin/sim get-archived-simulation <archiveid>

Simulation Factory Interactive Debugging

2011-03-11T20:30:01Z

Mthomas:

The Simulation Factory provides a simple facility to interactively debug Cactus/Einstein Toolkit simulation sessions in parallel.
Currently, only the Totalview debugger is supported.

== Supported Machines ==

Interactive parallel debugging using Totalview is currently supported on the following machines:

<ul>
<li>queenbee (LONI/TeraGrid)
<li>eric (LONI)
<li>ranger (TeraGrid)
<li>abe (TeraGrid)
<li> lincoln (TeraGrid)
<li> kraken (TeraGrid)
</ul>

== Getting Started ==

Before debugging, a Cactus/Einstein Toolkit binary must be compiled, and then a Simulation Factory simulation must be created. Instructions to do these steps can be found here:

[https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial Simulation_Factory_Advanced_Tutoria]

A second requirement is the existence of an Xwindows environment such as X11 or X-Ming.

== Interactive Sessions ==

To begin debugging, first launch an interactive session using The Simulation Factory. This can be done with the following command:

bin/sim interactive --walltime=01:00:00 --procs=8

walltime and procs can be changed to suit the amount of simulation time and processors needed.

== Interactive Debugging ==

Once the queuing system grants an interactive session, it will present a shell on a compute node. From here, it is
necessary to change the current working directory back to the simulation factory directory. Once located in The Simulation
Factory directory, an interactive debugging session can be launched by typing the following command:

bin/sim run-debug <simulationname>

For instance, if your simulation is called "test_sim", it would be:

bin/sim run-debug test_sim

Once the interactive debugging session launches. It is necessary to hit the "Go" button inside Totalview to begin execution of the Cactus/Einstein Toolkit simulation.

Simulation Factory Interactive Debugging

2011-03-11T19:55:45Z

Mthomas:

The Simulation Factory provides a simple facility to interactively debug Cactus/Einstein Toolkit simulation sessions in parallel.
Currently, only the Totalview debugger is supported.

== Supported Machines ==

Interactive parallel debugging using Totalview is currently supported on the following machines:

<ul>
<li>queenbee (LONI/TeraGrid)
<li>eric (LONI)
<li>ranger (TeraGrid)
<li>abe (TeraGrid)
<li> lincoln (TeraGrid)
<li> kraken (TeraGrid)
</ul>

== Getting Started ==

Before debugging, a Cactus/Einstein Toolkit binary must be compiled, and then a Simulation Factory simulation must be created. Instructions to do these steps can be found here:

[https://docs.einsteintoolkit.org/et-docs/Simulation_Factory_Advanced_Tutorial]

A second requirement is the existence of an Xwindows environment such as X11 or X-Ming.

== Interactive Sessions ==

To begin debugging, first launch an interactive session using The Simulation Factory. This can be done with the following command:

bin/sim interactive --walltime=01:00:00 --procs=8

walltime and procs can be changed to suit the amount of simulation time and processors needed.

== Interactive Debugging ==

Once the queuing system grants an interactive session, it will present a shell on a compute node. From here, it is
necessary to change the current working directory back to the simulation factory directory. Once located in The Simulation
Factory directory, an interactive debugging session can be launched by typing the following command:

bin/sim run-debug <simulationname>

For instance, if your simulation is called "test_sim", it would be:

bin/sim run-debug test_sim

Once the interactive debugging session launches. It is necessary to hit the "Go" button inside Totalview to begin execution of the Cactus/Einstein Toolkit simulation.

Pysim Tutorial for New Users

2010-12-06T19:21:08Z

Mthomas:

This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.

One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.

The example worked through below assumes that you are using the NSF TeraGrid resource [http://www.loni.org/systems/system.php?system=QueenBee Queen Bee]. If you don't have an existing account on Queen Bee you can [http://einsteintoolkit.org/info/documentation/request request an account] to use for these examples.

==Download, Configure, and Build==

Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):

./GetComponents -a http://www.simfactory.org/einsteintoolkit-pysim.th

This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

===Configure the Python Simulation Factory===

cd Cactus
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
vi simfactory/etc/defs.local.ini

Edit defs.local.ini and find the [default] section. Replace
<ul>
<li>YOUR_LOGIN with your username on Queen Bee
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your project allocation on Queen Bee
</ul>

===Compile Executable===

./simfactory/sim build --thornlist=manifest/einsteintoolkit.th

This may take a few minutes. You may receive an error message that building the utilities
failed. This is fine as long as the main executable has been built correctly. You can check this with the command <tt>ls exe</tt>; if this shows a file <tt>cactus_sim</tt>, then you are fine.)

==Run a Simulation==

From the Cactus directory, create a batch queue script and submit it to queue an example simulation:

./simfactory/sim submit static_tov --parfile=par/static_tov.par --procs=32 --walltime=8:0:0

This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.

To check the status of the simulation, use

./simfactory/sim list-simulations

==Look at Results==

For the following make sure that you have logged in to Queen Bee with X-forwarding enabled, i.e.

ssh -Y qb.loni.org

When the simulation is complete, move to the output directory. On Queen Bee
(qb.loni.org) the command will be (replace <user> with your own login name):

cd /scratch/<user>/simulations/static_tov/output-0001/static_tov

You should see a number of files with the extenstion ''.asc''. These are 0-D
(reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that
can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase::rho.maximum.asc). Start gnuplot with the command:

gnuplot

and at the gnuplot prompt type:

p 'hydrobase::rho.maximum.asc' u 2:3 w lp

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

[[Image:rho_of_time.png]]

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase::lapse.minimum.asc' u 2:3 w lp

[[Image:Lapse_of_time.png]]

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

==Additional Example Parameter Files==

In addition to the TOV star parameter file, we also provide a couple of vacuum parameter files: ks-mclachlan.par and qc0-mclachlan.par.
The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:

./simfactory/sim create-submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=32 --walltime=8:0:0

and

./simfactory/sim create-submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=32 --walltime=8:0:0

==Additional Simulation Factory Configuration==

The example worked out above works out of the box for the Queen Bee machine.
You may also be interested though in configuring the Simulation Factory for your
own machine or other Teragrid machines. In this section we provide a quick
example of what need to be changed or considered in simfactory/etc/udb.ini
in order to configure Simulation Factory for your needs. More specifically
we are going to work out an example for the TeraGrid machine Ranger.

The first step is to set your username for the machine. This means
to modify simfactory/etc/udb.ini customization section to reflect

'YOUR_LOGIN' on Ranger, for example:

[ranger]
user = YOUR_LOGIN

You also need to provide your allocation. For example, if your allocation
on Ranger is TG-PHY0123456N, underneath the [ranger] entry you just added, enter:

allocation = TG-PHY0123456N

Each machine has different policies on which file system to use to build
your application and run your simulation or how the file system tree
is organized. Simulation Factory keeps a database for several different
machines. You can peek at the directory simfactory/etc/mdb to have an idea of the list
of machines and what the default configuration is for each of them.
In particular for Ranger, the configuration for the source code path and
the simulation path defaults to '/work/00507/@USER@' and
'/scratch/00507/@USER@/simulations' respectively. You may need to
change it to reflect the group number (00507 here) you belong to. Once again,
underneath the [ranger] section previously added, set the following keys:

sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

After these changes, your entire [ranger] entry will look like the following:

[ranger]
user = YOUR_LOGIN
allocation = TG-PHY0123456N
sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

if your group number is '05555'. With these basic changes to
simfactory/etc/udb.ini, you are all set to configure and build the
Einstein Toolkit application on Ranger as well.

===New machines===

Instead of using one of the existing configurations you might want to add
a configuration for a new machine. the simfactory/etc/mdb/ directory contains
an example for such an entry: generic.ini. To configure this entry, copy the generic.ini
file to a new file named after your local machine. For a machine named varia, you would do:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/varia.ini

where you can replace 'varia' with your machine name. Once this is done, edit this file with your favorite text editor and
set various options (nickname, hostname, user, basedir, etc.) for this new
machine as above. The generic configuration is supported by a generic optionlist (see <tt> simfactory/etc/optionlists/generic.cfg </tt>).
These examples require several settings unavailable in that default configuration. You will need to customize an optionlist and set it, either in simfactory/etc/udb.ini as

[varia]
optionlist = optionlistname.cfg

or on the command-line; see the [https://svn.cct.lsu.edu/repos/numrel/simfactory/doc/simfactory.pdf simfactory documentation] for more details.

If the machine you added is a bigger resource (e.g. more than your personal
laptop or workstation), or can serve as example for some particular machine type,
we are happy to include its configuration in the main machine database.

==Any Problems?==

This example should work smoothly on the TeraGrid machines Queen Bee, Ranger, Kraken and Abe. If you have any problems, please send details to [mailto:users@einsteintoolkit.org users@einsteintoolkit.org].

Pysim Tutorial for New Users

2010-12-06T19:20:50Z

Mthomas:

This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.

One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.

The example worked through below assumes that you are using the NSF TeraGrid resource [http://www.loni.org/systems/system.php?system=QueenBee Queen Bee]. If you don't have an existing account on Queen Bee you can [http://einsteintoolkit.org/info/documentation/request request an account] to use for these examples.

s7I55J <a href="http://iazdlqonzhxm.com/">iazdlqonzhxm</a>, [url=http://ptfrihxblxqv.com/]ptfrihxblxqv[/url], [link=http://fzdmkjmnrpbf.com/]fzdmkjmnrpbf[/link], http://mnmyhypjmqip.com/

==Download, Configure, and Build==

Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):

./GetComponents -a http://www.simfactory.org/einsteintoolkit-pysim.th

This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

===Configure the Python Simulation Factory===

cd Cactus
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
vi simfactory/etc/defs.local.ini

Edit defs.local.ini and find the [default] section. Replace
<ul>
<li>YOUR_LOGIN with your username on Queen Bee
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your project allocation on Queen Bee
</ul>

===Compile Executable===

./simfactory/sim build --thornlist=manifest/einsteintoolkit.th

This may take a few minutes. You may receive an error message that building the utilities
failed. This is fine as long as the main executable has been built correctly. You can check this with the command <tt>ls exe</tt>; if this shows a file <tt>cactus_sim</tt>, then you are fine.)

==Run a Simulation==

From the Cactus directory, create a batch queue script and submit it to queue an example simulation:

./simfactory/sim submit static_tov --parfile=par/static_tov.par --procs=32 --walltime=8:0:0

This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.

To check the status of the simulation, use

./simfactory/sim list-simulations

==Look at Results==

For the following make sure that you have logged in to Queen Bee with X-forwarding enabled, i.e.

ssh -Y qb.loni.org

When the simulation is complete, move to the output directory. On Queen Bee
(qb.loni.org) the command will be (replace <user> with your own login name):

cd /scratch/<user>/simulations/static_tov/output-0001/static_tov

You should see a number of files with the extenstion ''.asc''. These are 0-D
(reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that
can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase::rho.maximum.asc). Start gnuplot with the command:

gnuplot

and at the gnuplot prompt type:

p 'hydrobase::rho.maximum.asc' u 2:3 w lp

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

[[Image:rho_of_time.png]]

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase::lapse.minimum.asc' u 2:3 w lp

[[Image:Lapse_of_time.png]]

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

==Additional Example Parameter Files==

In addition to the TOV star parameter file, we also provide a couple of vacuum parameter files: ks-mclachlan.par and qc0-mclachlan.par.
The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:

./simfactory/sim create-submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=32 --walltime=8:0:0

and

./simfactory/sim create-submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=32 --walltime=8:0:0

==Additional Simulation Factory Configuration==

The example worked out above works out of the box for the Queen Bee machine.
You may also be interested though in configuring the Simulation Factory for your
own machine or other Teragrid machines. In this section we provide a quick
example of what need to be changed or considered in simfactory/etc/udb.ini
in order to configure Simulation Factory for your needs. More specifically
we are going to work out an example for the TeraGrid machine Ranger.

The first step is to set your username for the machine. This means
to modify simfactory/etc/udb.ini customization section to reflect

'YOUR_LOGIN' on Ranger, for example:

[ranger]
user = YOUR_LOGIN

You also need to provide your allocation. For example, if your allocation
on Ranger is TG-PHY0123456N, underneath the [ranger] entry you just added, enter:

allocation = TG-PHY0123456N

Each machine has different policies on which file system to use to build
your application and run your simulation or how the file system tree
is organized. Simulation Factory keeps a database for several different
machines. You can peek at the directory simfactory/etc/mdb to have an idea of the list
of machines and what the default configuration is for each of them.
In particular for Ranger, the configuration for the source code path and
the simulation path defaults to '/work/00507/@USER@' and
'/scratch/00507/@USER@/simulations' respectively. You may need to
change it to reflect the group number (00507 here) you belong to. Once again,
underneath the [ranger] section previously added, set the following keys:

sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

After these changes, your entire [ranger] entry will look like the following:

[ranger]
user = YOUR_LOGIN
allocation = TG-PHY0123456N
sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

if your group number is '05555'. With these basic changes to
simfactory/etc/udb.ini, you are all set to configure and build the
Einstein Toolkit application on Ranger as well.

===New machines===

Instead of using one of the existing configurations you might want to add
a configuration for a new machine. the simfactory/etc/mdb/ directory contains
an example for such an entry: generic.ini. To configure this entry, copy the generic.ini
file to a new file named after your local machine. For a machine named varia, you would do:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/varia.ini

where you can replace 'varia' with your machine name. Once this is done, edit this file with your favorite text editor and
set various options (nickname, hostname, user, basedir, etc.) for this new
machine as above. The generic configuration is supported by a generic optionlist (see <tt> simfactory/etc/optionlists/generic.cfg </tt>).
These examples require several settings unavailable in that default configuration. You will need to customize an optionlist and set it, either in simfactory/etc/udb.ini as

[varia]
optionlist = optionlistname.cfg

or on the command-line; see the [https://svn.cct.lsu.edu/repos/numrel/simfactory/doc/simfactory.pdf simfactory documentation] for more details.

If the machine you added is a bigger resource (e.g. more than your personal
laptop or workstation), or can serve as example for some particular machine type,
we are happy to include its configuration in the main machine database.

==Any Problems?==

This example should work smoothly on the TeraGrid machines Queen Bee, Ranger, Kraken and Abe. If you have any problems, please send details to [mailto:users@einsteintoolkit.org users@einsteintoolkit.org].

Simulation Factory Advanced Tutorial

2010-12-06T19:20:18Z

Mthomas:

The Simulation Factory simplifies many aspects of running Cactus-based
simulations. It provides a central facility for managing authoritative
source tree versions, providing convenient access to remote HPC
systems, building Cactus source configurations, and managing
simulations all the way from submission to archiving their output.

== Getting Started ==

To begin using The Simulation Factory, it needs to be checked out from
'''svn'''. The Simulation Factory is typically placed into a
'''simfactory''' folder inside a Cactus source tree. This can be
accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location
to be used with multiple Cactus source trees. This approach will be
described later.

== Initial Setup ==

Once the Simulation Factory has been checked out from svn, the next
step is to configure it, telling it e.g. about your user name. The
Simulation Factory comes with example configuration files that you can
copy and modify. Assuming The Simulation Factory has been checked out
into the '''simfactory''' folder, this initial configuration can be
accomplished with the following commands:

cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configurations ===

The Simulation Factory contains a database known as the Machine
Database. This collection of information describes all the aspects
that are unique about each individual HPC system, so that the
Simulation Factory can provide a common interface for all systems that
hides these differences.

The Machine Database consists of different sections, one for each
machine. The section name is given in square brackets, e.g.
'''[queenbee]'''. There is a special section '''[default]''' that
provides default values for those properties that are not explicitly
set in the machine-specific entries.

The Machine Database is an authoritative collection of information,
and is generally not meant to contain modification that are only
relevant for individual people. These local modifications are instead
maintained in the file '''simfactory/etc/defs.local.ini''' described
above, where one can add, change, or overwrite properties of Machine
Database entries. For instance, if an alternative username,
allocation, and sourcebasedir is needed for the machine
'''queenbee''', you would add the following section there:

[queenbee]
user = QUEENBEE_USERNAME
allocation = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries.
The most useful is probably '''@USER@''', which expands to the
'''user''' property of the Machine Database entry.

For example, if you are using the same user name on many systems, but
have a different user name on some systems, then you would set the
common user name in the '''[default]''' section, and override this for
those machines where your user name differs. The example
'''simfactory/etc/defs.local.ini.complex''' has examples for this.

Most of the macros available in the Simulation Factory are described
in the section [[#Macros]] below.

The command

simfactory/sim list-machines

outputs a list of all preconfigured machines that the Simulation
Factory knows about.

=== Local Workstation Configuration ===

The Simulation Factory can only be used on a machine known to the
Simulation Factory. This means that you may have to add an entry for
your workstation or notebook.

The first step is to determine the hostname of the local machine. It
is through this hostname that the Simulation Factory matches a Machine
Database entry to the machine on which it executes. The hostname can
be determined using the following command:

hostname

Once you know the hostname, issue the following command to create a
new Machine Database entry, starting from a generic example:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Then edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li>The section header for this machine database entry must be unique
among all Machine Database entries, and must match the '''nickname'''
property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root'''
path under which all your Cactus source trees reside. (This is not
the Cactus directory itself, but the directory that contains the
Cactus directory.)
<li> '''basedir = /home/@USER@/simulations''' with the desired folder
that will contain all simulation output. (This is explained in more
detail below in the section [[#Managing Simulations]].)
</ul>
You can ignore the properties '''user''', '''email''', and
'''allocation''', as the values from the '''[default]''' section of
'''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory simplifies access to remote systems, both for
transferring files and logging in. You can synchronise (replicate) an
authoritative version of your Cactus source tree to remote systems,
obtain an interactive shell, or execute commands.

=== Information Commands ===

The following commands can be used to discover information about a
machine, or list all known machines.

List all known machines:

simfactory/sim list-machines

List details about a single machine:

simfactory/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is
currently being executed:

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed
into a central location on each machine, but are instead built
on-demand by every user for a certain thorn list. (One of the
advantages is that people can thus easily add their own thorns.) To
help with this approach, the Simulation Factory provides a facility to
synchronize a Cactus user's local, authoritative source tree to remote
HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like
mechanisms such as gsi-ssh. Currently, you must still manage all ssh
keys and passwords manually. (We highly recommend to use ssh keychain
and ssh agents to avoid having to enter passwords multiple times.)

==== Configuration ====

Before syncing a source tree to a remote system, a small amount of
configuration must be performed. It is necessary to either verify that
the defaults are correct, or to define the correct values for the
following keys for the remote system in the Machine Database:

<ul>
<li> '''sourcebasedir'''
<ul>
<li>The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li>The root directory in which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li>The user name on the remote system
</ul>
</ul>

You can output the currently configured values by issuing the command

simfactory/sim print-mdb <machine>

If you need to change these values, then edit (on the local system)
the file '''simfactory/etc/defs.local.ini''' and add a section for the
remote machine. This entry will augment the existing Machine Database
entry and updating/replace the corresponding values. An example for
the machine '''queenbee''' can be see in the [[#Additional
Configuration]] section.

To see the list of files and directories that are synchronized, look
at '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''sync-sources'''
<ul>
<li>The list of files and directories that will be transferred when
the option '''--sync-sourcetree''' is enabled (on by default)
</ul>
<li> '''sync-parfiles'''
<ul>
<li>The list of files and directories that will be copied when the
option '''--sync-parfiles''' is enabled (also on by default). This
list of files typically includes just parameter files.
</ul>
<li> '''sync-excludes'''
<ul>
<li>A list of files and directories that will be expressly excluded
from syncing, such as e.g. '''CVS''' or '''.svn''' directories.
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two options, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li>Synchronise the complete source tree, as specified in the
aforementioned '''rsync-sources''' configuration entry. This takes a
few seconds or minutes, depending on the connection.
</ul>
<li> '''sync-parfiles'''
<ul>
<li>Synchronise parameter files, as specified by the aforementioned
'''rsync-parfiles''' configuration entry. This is typically faster
than synchronising the source tree.
</ul>
</ul>

Usually, you would issue the command:

simfactory/sim sync <machine>

To synchronise only parfiles, you can negate the
'''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from
another remote machine, then use

simfactory/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine
'''<frommachine>'''.

=== Remote Login ===

The Simulation Factory provides the ability to log in to a remote
system. This is initiated with the command

simfactory/sim login <machine>

This will automatically cd into the Cactus directory on the remote
system.

=== Local/Remote Command Execution ===

To execute a command (locally) via the Simulation Factory, use the
command

simfactory/sim execute <command>

The command will be executed in the Cactus directory on the remote
system.

If the command is complex, and requires arguments, the command must be
quoted. For example

simfactory/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

== Build Cactus Configurations ==

The Simulation Factory provides a central facility for configuring and
building Cactus source trees. When a Cactus source tree is compiled,
the Simulation Factory creates a '''configuration''' for the compiled
executable, storing with it related information such as the Cactus
options list, and the scripts necessary to submit and run jobs in a
queuing system. This configuration is thus a self-contained entity
containing everything that is necessary to perform Cactus simulations.

=== Information Commands ===

To list all existing Cactus configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

To build a configuration, four pieces of information are required:

<ul>
<li>Thorn list
<ul>
<li>This defines which thorns are to be included into the configuration.
<li>Default: '''thornlist''' parameter of the Machine Database entry
<li>Override: '''--thornlist=<thornlist>'''
<li>The default option list is probably not useful in many cases.
</ul>
<li>Option List
<ul>
<li>This specifies the compiler and build options that need to be used
to build Cactus on a particular system.
<li>Default: '''optionlist''' parameter of the Machine Database entry
<li>Override: '''--optionlist=<optionlist>'''
<li>The Simulation Factory is supposed to contain good, working
default option lists for all supported systems. In fact, this is one
of the main strengths of the Simulation Factory. You should normally
not need to override the default.
</ul>
<li>Submission Script
<ul>
<li>This specifies how to submit a job to the queueing system on a
particular system.
<li>Default: '''submitscript''' parameter of the Machine Database entry
<li>Override: '''--submitscript=<submitscript>'''
<li>Similar to the option list, the Simulation Factory is supposed to
contain good, working default submission scripts for all supported
systems.
</ul>
<li>Run Script
<ul>
<li>This specifies to how execute an MPI process on a particular
system; it is closely connected to the submission script.
<li>Default: '''runscript''' parameter of the Machine Database entry
<li>Override: '''--runscript=<runscript>'''
<li>Same as with the the submission script, the Simulation Factory is
supposed to contain good, working default run scripts for all
supported systems.
</ul>
</ul>

To build a configuration with a specific thornlist, issue the
following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to
'sim'. (We recommend this.) You can in addition specify any of the
options
<ul>
<li>--debug
<li>--optimise (default)
<li>--profile
</ul>
These options create configurations for debugging, optimisation (this
is the default), or profiling enabled.
If any of these options is specified, then the configuration name will
be modified correspondingly, e.g. to 'sim-debug'.

==== Additional Options ====

<ul>

<li>'''--reconfig'''
<ul>
<li>Reconfigure before building, i.e. re-examine the configuration
options and re-run the CST stage. This happens automatically when
the option list changes.
</ul>

<li>'''--clean'''
<ul>
<li>Clean the configuration (remove all object files etc.) before
building.
</ul>

</ul>

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

=== Script Locations ===

The Simulation Factory provides default scripts for all its
preconfigured machines. These scripts can be found in the following
locations

<ul>

<li>'''Option Lists'''
<ul>
<li>MDB Key: optionlist
<li>Location: simfactory/etc/optionlists
</ul>

<li>'''Submit Scripts'''
<ul>
<li>MDB Key: submitscript
<li>Location: simfactory/etc/submitscripts
</ul>

<li>'''Run Scripts'''
<ul>
<li>MDB Key: runscript
<li>Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list Queen Bee uses by
default, issue the command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for
submitting, running, and managing simulations. This is accomplished
through two main commands '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations on a particular machine can be seen with
the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose
option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a
simulation to the host queuing system. They are

<ul>

<li>'''Configuration'''
<ul>
<li>The Cactus configuration to run
<li>'''option''': --configuration
<li>'''default''': "sim"
</ul>

<li>'''Parfile'''
<ul>
<li>The Cactus parameter file to use
<li>'''option''': --parfile
</ul>

<li>'''Walltime'''
<ul>
<li>The total amount of wall time required
<li>'''option''': --walltime
<li>'''default''': MDB Key '''maxwalltime'''
</ul>

<li>Processors
<ul>
<li>The total number of processors to use
<li>'''option''': --procs
<li>'''default''': 1
</ul>

</ul>

The option '''--configuration''' only needs to be specified the first
time you submit a simulation. Subsequent re-submissions of the same
simulation (for restarting from checkpoints) will always use the same
configuration that was specified the first time. Here is an example of
submitting a simulation named "static_tov" using the aforementioned
options:

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where
you do not need to specify the option names, but have to specify the
options in a certain order. If you don't specify a simulation name
using the shorthand syntax, a simulation name will be derived from
from the parameter file name.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li>Number of OpenMP Threads
<ul>
<li>The number of OpenMP threads per MPI process. (You specify the
total number of processors (cores), and the number of OpenMP
threads; the number of MPI processes is then calculated
automatically.)
<li>option: --num-threads
<li>default: 1 (as if OpenMP was not used)
</ul>

<li>Allocation
<ul>
<li>The allocation for the simulation, overriding the corresponding
MDB entry
<li>option: --allocation
<li>default: taken from the MDB
</ul>

<li>Queue
<ul>
<li>The queue for the simulation, overriding the corresponding MDB
entry
<li>option: --queue
<li>default: taken from the MDB
</ul>

<li>Processors per node
<ul>
<li>The number of processors per node requested from the queueing system
<li>option: --ppn
<li>default: all processors on a node
</ul>

<li>Used processors per node
<ul>
<li>The number of processors per node that should actually be used,
allowing under-using nodes even if the queueing system does not
allow it. (The remaining processors will idle and will remain
unused.)
<li>option: --ppn-used
<li>default: all processors on a node
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing
the queuing system. Running a simulation directly uses the same
options, but ignores wall time limit etc. You use the '''run'''
command for this:

simfactory/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the command

simfactory/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise
similar to the submit command.

To stop a simulation:

simfactory/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is submitted for the first time, all necessary
information from the Cactus build configuration is brought into a
specific simulation folder created underneath the '''basedir'''
directory. Contained inside this folder, which has the same name as
the specified simulation, are the executable, run script, submit
script, a SIMFACTORY folder, a log file, and the output directories
for each individual restart.

Simulations are self-contained, and once created do not rely on
outside information. For example, recompiling the executable or
changing the parameter file that were used to submit a simulation will
not influence the simulation, since the simulation contains copies of
both. This ensures that simulations can continue to run unperturbed
even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with
several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script
files for submission and execution, and a properties.ini file that is
used by the Simulation Factory to store information about the
simulation.

Each time a simulation is either run or submitted, a restart directory
is created underneath the simulation directory. This restart folder
has a name of the format "output-####", starting with "output-0000".
Contained inside the restart folder are several internal files, the
output written to stdout and stderr from the simulation, and the
simulation output itself. The simulation output is typically stored
inside a directory named after the basename of the parameter file. An
example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run
script from the build configuration into the folder
"basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/"
folder, the run and submit scripts into the "run/" folder, the Cactus
options list into the "cfg/" folder, and the parfile into the "par/"
folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to the
Simulation Factory using the PetaShare data storage system on LONI.
This archiving system can only be used from machines that have access
to PetaShare. The currently supported machines that have access to
Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare
[http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz
Pcommands],
[http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php
(manual)] must be downloaded and installed onto your machine. Once
that is done, add a new section in '''etc/defs.local.ini''' for the
machine from which you wish to access PetaShare, and add the necessary
MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

'''archivebasepath''' is where the simulations will be stored on
PetaShare.

Once this setup has been completed, you can archive an entire
simulation (including all restarts) using the command

simfactory/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the
command

simfactory/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/sim list-archived-simulations

To retreive an archived simulation, first use the
'''list-archived-simulations''' command to retrieve the unique
identifier for the simulation, and then issue the following command to
retrieve a simulation. It will place the simulation in the current
directory.

simfactory/sim get-archived-simulation <archiveid>

Simulation Factory Advanced Tutorial

2010-11-29T15:33:02Z

Mthomas:

The Simulation Factory simplifies many aspects of running Cactus-based
simulations. It provides a central facility for managing authoritative
source tree versions, providing convenient access to remote HPC
systems, building Cactus source configurations, and managing
simulations all the way from submission to archiving their output.

== Getting Started ==

To begin using The Simulation Factory, it needs to be checked out from
'''svn'''. The Simulation Factory is typically placed into a
'''simfactory''' folder inside a Cactus source tree. This can be
accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory could also be placed in an independent location
to be used with multiple Cactus source trees. This approach will be
described later.

== Initial Setup ==

Once the Simulation Factory has been checked out from svn, the next
step is to configure it, telling it e.g. about your user name. The
Simulation Factory comes with example configuration files that you can
copy and modify. Assuming The Simulation Factory has been checked out
into the '''simfactory''' folder, this initial configuration can be
accomplished with the following commands:

cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configurations ===

The Simulation Factory contains a database known as the Machine
Database. This collection of information describes all the aspects
that are unique about each individual HPC system, so that the
Simulation Factory can provide a common interface for all systems that
hides these differences.

The Machine Database consists of different sections, one for each
machine. The section name is given in square brackets, e.g.
'''[queenbee]'''. There is a special section '''[default]''' that
provides default values for those properties that are not explicitly
set in the machine-specific entries.

The Machine Database is an authoritative collection of information,
and is generally not meant to contain modification that are only
relevant for individual people. These local modifications are instead
maintained in the file '''simfactory/etc/defs.local.ini''' described
above, where one can add, change, or overwrite properties of Machine
Database entries. For instance, if an alternative username,
allocation, and sourcebasedir is needed for the machine
'''queenbee''', you would add the following section there:

[queenbee]
user = QUEENBEE_USERNAME
allocation = QUEENBEE_ALLOCATION
sourcebasedir = /work/@USER@

There are several macros that help simplifying configuration entries.
The most useful is probably '''@USER@''', which expands to the
'''user''' property of the Machine Database entry.

For example, if you are using the same user name on many systems, but
have a different user name on some systems, then you would set the
common user name in the '''[default]''' section, and override this for
those machines where your user name differs. The example
'''simfactory/etc/defs.local.ini.complex''' has examples for this.

Most of the macros available in the Simulation Factory are described
in the section [[#Macros]] below.

The command

simfactory/sim list-machines

outputs a list of all preconfigured machines that the Simulation
Factory knows about.

=== Local Workstation Configuration ===

The Simulation Factory can only be used on a machine known to the
Simulation Factory. This means that you may have to add an entry for
your workstation or notebook.

The first step is to determine the hostname of the local machine. It
is through this hostname that the Simulation Factory matches a Machine
Database entry to the machine on which it executes. The hostname can
be determined using the following command:

hostname

Once you know the hostname, issue the following command to create a
new Machine Database entry, starting from a generic example:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Then edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li>The section header for this machine database entry must be unique
among all Machine Database entries, and must match the '''nickname'''
property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root'''
path under which all your Cactus source trees reside. (This is not
the Cactus directory itself, but the directory that contains the
Cactus directory.)
<li> '''basedir = /home/@USER@/simulations''' with the desired folder
that will contain all simulation output. (This is explained in more
detail below in the section [[#Managing Simulations]].)
</ul>
You can ignore the properties '''user''', '''email''', and
'''allocation''', as the values from the '''[default]''' section of
'''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory simplifies access to remote systems, both for
transferring files and logging in. You can synchronise (replicate) an
authoritative version of your Cactus source tree to remote systems,
obtain an interactive shell, or execute commands.

=== Information Commands ===

The following commands can be used to discover information about a
machine, or list all known machines.

List all known machines:

simfactory/sim list-machines

List details about a single machine:

simfactory/sim list-machine <machine>

Print the complete Machine Database to the screen:

simfactory/sim print-mdb

Print the Machine Database entry for a single machine:

simfactory/sim print-mdb <machine>

Print the name of the machine on which the Simulation Factory is
currently being executed:

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed
into a central location on each machine, but are instead built
on-demand by every user for a certain thorn list. (One of the
advantages is that people can thus easily add their own thorns.) To
help with this approach, the Simulation Factory provides a facility to
synchronize a Cactus user's local, authoritative source tree to remote
HPC systems, where it can then be compiled and run.

Remote access is implemented on top of ssh and other ssh-like
mechanisms such as gsi-ssh. Currently, you must still manage all ssh
keys and passwords manually. (We highly recommend to use ssh keychain
and ssh agents to avoid having to enter passwords multiple times.)

==== Configuration ====

Before syncing a source tree to a remote system, a small amount of
configuration must be performed. It is necessary to either verify that
the defaults are correct, or to define the correct values for the
following keys for the remote system in the Machine Database:

<ul>
<li> '''sourcebasedir'''
<ul>
<li>The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li>The root directory in which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li>The user name on the remote system
</ul>
</ul>

You can output the currently configured values by issuing the command

simfactory/sim print-mdb <machine>

If you need to change these values, then edit (on the local system)
the file '''simfactory/etc/defs.local.ini''' and add a section for the
remote machine. This entry will augment the existing Machine Database
entry and updating/replace the corresponding values. An example for
the machine '''queenbee''' can be see in the [[#Additional
Configuration]] section.

To see the list of files and directories that are synchronized, look
at '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''sync-sources'''
<ul>
<li>The list of files and directories that will be transferred when
the option '''--sync-sourcetree''' is enabled (on by default)
</ul>
<li> '''sync-parfiles'''
<ul>
<li>The list of files and directories that will be copied when the
option '''--sync-parfiles''' is enabled (also on by default). This
list of files typically includes just parameter files.
</ul>
<li> '''sync-excludes'''
<ul>
<li>A list of files and directories that will be expressly excluded
from syncing, such as e.g. '''CVS''' or '''.svn''' directories.
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two options, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li>Synchronise the complete source tree, as specified in the
aforementioned '''rsync-sources''' configuration entry. This takes a
few seconds or minutes, depending on the connection.
</ul>
<li> '''sync-parfiles'''
<ul>
<li>Synchronise parameter files, as specified by the aforementioned
'''rsync-parfiles''' configuration entry. This is typically faster
than synchronising the source tree.
</ul>
</ul>

Usually, you would issue the command:

simfactory/sim sync <machine>

To synchronise only parfiles, you can negate the
'''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If you want to synchronise not from the local machine, but from
another remote machine, then use

simfactory/sim --remote=<frommachine> sync <tomachine>

This executes the synchronisation command on the machine
'''<frommachine>'''.

=== Remote Login ===

The Simulation Factory provides the ability to log in to a remote
system. This is initiated with the command

simfactory/sim login <machine>

This will automatically cd into the Cactus directory on the remote
system.

=== Local/Remote Command Execution ===

To execute a command (locally) via the Simulation Factory, use the
command

simfactory/sim execute <command>

The command will be executed in the Cactus directory on the remote
system.

If the command is complex, and requires arguments, the command must be
quoted. For example

simfactory/sim execute 'ls -al'

To execute a remote command, use the command

simfactory/sim --remote-machine=<machine> execute <command>

An example of a complex command being executed remotely is

simfactory/sim --remote-machine=queenbee execute 'find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;'

== Build Cactus Configurations ==

The Simulation Factory provides a central facility for configuring and
building Cactus source trees. When a Cactus source tree is compiled,
the Simulation Factory creates a '''configuration''' for the compiled
executable, storing with it related information such as the Cactus
options list, and the scripts necessary to submit and run jobs in a
queuing system. This configuration is thus a self-contained entity
containing everything that is necessary to perform Cactus simulations.

=== Information Commands ===

To list all existing Cactus configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

To build a configuration, four pieces of information are required:

<ul>
<li>Thorn list
<ul>
<li>This defines which thorns are to be included into the configuration.
<li>Default: '''thornlist''' parameter of the Machine Database entry
<li>Override: '''--thornlist=<thornlist>'''
<li>The default option list is probably not useful in many cases.
</ul>
<li>Option List
<ul>
<li>This specifies the compiler and build options that need to be used
to build Cactus on a particular system.
<li>Default: '''optionlist''' parameter of the Machine Database entry
<li>Override: '''--optionlist=<optionlist>'''
<li>The Simulation Factory is supposed to contain good, working
default option lists for all supported systems. In fact, this is one
of the main strengths of the Simulation Factory. You should normally
not need to override the default.
</ul>
<li>Submission Script
<ul>
<li>This specifies how to submit a job to the queueing system on a
particular system.
<li>Default: '''submitscript''' parameter of the Machine Database entry
<li>Override: '''--submitscript=<submitscript>'''
<li>Similar to the option list, the Simulation Factory is supposed to
contain good, working default submission scripts for all supported
systems.
</ul>
<li>Run Script
<ul>
<li>This specifies to how execute an MPI process on a particular
system; it is closely connected to the submission script.
<li>Default: '''runscript''' parameter of the Machine Database entry
<li>Override: '''--runscript=<runscript>'''
<li>Same as with the the submission script, the Simulation Factory is
supposed to contain good, working default run scripts for all
supported systems.
</ul>
</ul>

To build a configuration with a specific thornlist, issue the
following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to
'sim'. (We recommend this.) You can in addition specify any of the
options
<ul>
<li>--debug
<li>--optimise (default)
<li>--profile
</ul>
These options create configurations for debugging, optimisation (this
is the default), or profiling enabled.
If any of these options is specified, then the configuration name will
be modified correspondingly, e.g. to 'sim-debug'.

==== Additional Options ====

<ul>

<li>'''--reconfig'''
<ul>
<li>Reconfigure before building, i.e. re-examine the configuration
options and re-run the CST stage. This happens automatically when
the option list changes.
</ul>

<li>'''--clean'''
<ul>
<li>Clean the configuration (remove all object files etc.) before
building.
</ul>

</ul>

cbZsul <a href="http://wlswjasvvkxf.com/">wlswjasvvkxf</a>, [url=http://aegknafsayjg.com/]aegknafsayjg[/url], [link=http://erlxqmjagmjm.com/]erlxqmjagmjm[/link], http://rbnzmidhcrnk.com/

=== Script Locations ===

The Simulation Factory provides default scripts for all its
preconfigured machines. These scripts can be found in the following
locations

<ul>

<li>'''Option Lists'''
<ul>
<li>MDB Key: optionlist
<li>Location: simfactory/etc/optionlists
</ul>

<li>'''Submit Scripts'''
<ul>
<li>MDB Key: submitscript
<li>Location: simfactory/etc/submitscripts
</ul>

<li>'''Run Scripts'''
<ul>
<li>MDB Key: runscript
<li>Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list Queen Bee uses by
default, issue the command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for
submitting, running, and managing simulations. This is accomplished
through two main commands '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations on a particular machine can be seen with
the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose
option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a
simulation to the host queuing system. They are

<ul>

<li>'''Configuration'''
<ul>
<li>The Cactus configuration to run
<li>'''option''': --configuration
<li>'''default''': "sim"
</ul>

<li>'''Parfile'''
<ul>
<li>The Cactus parameter file to use
<li>'''option''': --parfile
</ul>

<li>'''Walltime'''
<ul>
<li>The total amount of wall time required
<li>'''option''': --walltime
<li>'''default''': MDB Key '''maxwalltime'''
</ul>

<li>Processors
<ul>
<li>The total number of processors to use
<li>'''option''': --procs
<li>'''default''': 1
</ul>

</ul>

The option '''--configuration''' only needs to be specified the first
time you submit a simulation. Subsequent re-submissions of the same
simulation (for restarting from checkpoints) will always use the same
configuration that was specified the first time. Here is an example of
submitting a simulation named "static_tov" using the aforementioned
options:

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand notation where
you do not need to specify the option names, but have to specify the
options in a certain order. If you don't specify a simulation name
using the shorthand syntax, a simulation name will be derived from
from the parameter file name.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example is

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li>Number of OpenMP Threads
<ul>
<li>The number of OpenMP threads per MPI process. (You specify the
total number of processors (cores), and the number of OpenMP
threads; the number of MPI processes is then calculated
automatically.)
<li>option: --num-threads
<li>default: 1 (as if OpenMP was not used)
</ul>

<li>Allocation
<ul>
<li>The allocation for the simulation, overriding the corresponding
MDB entry
<li>option: --allocation
<li>default: taken from the MDB
</ul>

<li>Queue
<ul>
<li>The queue for the simulation, overriding the corresponding MDB
entry
<li>option: --queue
<li>default: taken from the MDB
</ul>

<li>Processors per node
<ul>
<li>The number of processors per node requested from the queueing system
<li>option: --ppn
<li>default: all processors on a node
</ul>

<li>Used processors per node
<ul>
<li>The number of processors per node that should actually be used,
allowing under-using nodes even if the queueing system does not
allow it. (The remaining processors will idle and will remain
unused.)
<li>option: --ppn-used
<li>default: all processors on a node
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing
the queuing system. Running a simulation directly uses the same
options, but ignores wall time limit etc. You use the '''run'''
command for this:

simfactory/sim run static_tov --configuration sim-debug --parfile=par/static_tov.par --procs=8

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the command

simfactory/sim interactive --procs=8 --walltime=4:00:00

This leads to a login shell on the compute node, but is otherwise
similar to the submit command.

To stop a simulation:

simfactory/sim stop <simulationname>

To purge (put in the basedir/TRASH folder) an existing simulation:

simfactory/sim purge <simulationname> [--restart-id=<restartid>]

To show the current output (stdout and stderr) for a given simulation:

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is submitted for the first time, all necessary
information from the Cactus build configuration is brought into a
specific simulation folder created underneath the '''basedir'''
directory. Contained inside this folder, which has the same name as
the specified simulation, are the executable, run script, submit
script, a SIMFACTORY folder, a log file, and the output directories
for each individual restart.

Simulations are self-contained, and once created do not rely on
outside information. For example, recompiling the executable or
changing the parameter file that were used to submit a simulation will
not influence the simulation, since the simulation contains copies of
both. This ensures that simulations can continue to run unperturbed
even weeks after they have been created.

Here is the contents of the simulation folder "static_tov" with
several restarts in it:

[mwt@eric2 simulations]$ ls -l static_tov
total 32
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0000
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0002
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0003
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0004
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script
files for submission and execution, and a properties.ini file that is
used by the Simulation Factory to store information about the
simulation.

Each time a simulation is either run or submitted, a restart directory
is created underneath the simulation directory. This restart folder
has a name of the format "output-####", starting with "output-0000".
Contained inside the restart folder are several internal files, the
output written to stdout and stderr from the simulation, and the
simulation output itself. The simulation output is typically stored
inside a directory named after the basename of the parameter file. An
example output directory is:

[mwt@eric2 output-0001]$ ls -l
total 172
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run
script from the build configuration into the folder
"basedir/<simulation>/SIMFACTORY". The executable goes in the "exe/"
folder, the run and submit scripts into the "run/" folder, the Cactus
options list into the "cfg/" folder, and the parfile into the "par/"
folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -lR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to the
Simulation Factory using the PetaShare data storage system on LONI.
This archiving system can only be used from machines that have access
to PetaShare. The currently supported machines that have access to
Petashare are

bluedawg
ducky
eric
lacumba
louie
neptune
oliver
painter
poseidon
queenbee
spider
tezpur
zeke

To use the PetaShare archiving system, the PetaShare
[http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz
Pcommands],
[http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php
(manual)] must be downloaded and installed onto your machine. Once
that is done, add a new section in '''etc/defs.local.ini''' for the
machine from which you wish to access PetaShare, and add the necessary
MDB keys. An example for the machine Queen Bee is:

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

'''archivebasepath''' is where the simulations will be stored on
PetaShare.

Once this setup has been completed, you can archive an entire
simulation (including all restarts) using the command

simfactory/sim archive <simulationname>

To archive just a single restart for a given simulation, issue the
command

simfactory/sim archive <simulationname> --restart-id=<restartid>

To print a list of all archived simulations, use the command

simfactory/sim list-archived-simulations

To retreive an archived simulation, first use the
'''list-archived-simulations''' command to retrieve the unique
identifier for the simulation, and then issue the following command to
retrieve a simulation. It will place the simulation in the current
directory.

simfactory/sim get-archived-simulation <archiveid>

Simulation Factory Advanced Tutorial

2010-09-30T19:13:24Z

Mthomas: /* Archiving */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is run for the first time, all the necessary information from the Cactus build configuration is brought into a specific simulation folder created
underneath the '''basedir'''. Contained inside this folder, which has the same name as the specified simulation, are the executable, the run script, the submit script, the SIMFACTORY folder, a log file, and the output directories of each individual restart.

Here is the contents of the simulation folder "btest" with several restarts in it

[mwt@eric2 simulations]$ ls -al btest
total 32
drwxr-xr-x 8 mwt lsuusers 4096 Sep 17 09:03 .
drwxr-xr-x 8 mwt lsuusers 4096 Sep 27 11:32 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0002
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0003
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0004
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0005
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script files needed for submitting and execution, and a properties.ini file that is used by the Simulation Factory to store
information about the simulation.

Each time a simulation is either run or submitted, a restart directory is created underneath the simulation directory. This restart folder has the format of output-####, starting with
output-0001. Contained inside the restart folder are several internal files, the output written to stdout and stderr from the simulation, and the simulation output itself. The simulation output is stored inside a directory named after the basename of the parameter file. An example output directory is below

[mwt@eric2 output-0001]$ ls -al
total 172
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 21:06 .
drwxr-xr-x 4 mwt lsuusers 4096 Sep 27 11:29 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run script from the build configuration into the basedir/<simulation>/SIMFACTORY folder. The executable goes in the exe/ folder, the run and submit scripts into the run/ folder, the Cactus options list into the cfg/ folder, and the parfile into the par/ folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -alR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to The Simulation Factory using the Petashare data storage system. This archiving system can only be used from machines that have access to Petashare. The currently supported machines that have access to Petashare are

queenbee
tezpur
eric
ducky
neptune
zeke
spider
oliver
lacumba
painter
bluedawg
poseidon
louie

In order to use the Petashare archiving system, the Petashare [http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz Pcommands], [http://www.cct.lsu.edu/~sreekanth/petashare/pcommands-manual(2.0).php (manual)] must be downloaded and installed into your local directory. Once that's done, add a new section in '''etc/defs.local.ini''' for the machine you wish to access petashare from, and add the necessary keys. An example for the machine queenbee is below

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

archivebasepath is where the simulations will be stored on petashare.

Once setup has been completed, you can archive an entire simulation (including all restarts) using the following command

simfactory/sim archive <simulationname>

To archive just a single restart for a given simulation, you can issue the following command

simfactory/sim archive <simulationname> -- restart-id=<restartid>

To get a list of all archived simulations, use the following command

simfactory/sim list-archived-simulations

To retreive an archived simulation, first use the '''list-archived-simulations''' command to retrieve the unique identifier for the simulation, and then issue the following command to retrieve a simulation. It will place the simulation in the current directory.

simfactory/sim get-archived-simulation <archiveid>

=== Macros ===

Simulation Factory Advanced Tutorial

2010-09-30T19:11:20Z

Mthomas: /* Archiving */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is run for the first time, all the necessary information from the Cactus build configuration is brought into a specific simulation folder created
underneath the '''basedir'''. Contained inside this folder, which has the same name as the specified simulation, are the executable, the run script, the submit script, the SIMFACTORY folder, a log file, and the output directories of each individual restart.

Here is the contents of the simulation folder "btest" with several restarts in it

[mwt@eric2 simulations]$ ls -al btest
total 32
drwxr-xr-x 8 mwt lsuusers 4096 Sep 17 09:03 .
drwxr-xr-x 8 mwt lsuusers 4096 Sep 27 11:32 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0002
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0003
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0004
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0005
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script files needed for submitting and execution, and a properties.ini file that is used by the Simulation Factory to store
information about the simulation.

Each time a simulation is either run or submitted, a restart directory is created underneath the simulation directory. This restart folder has the format of output-####, starting with
output-0001. Contained inside the restart folder are several internal files, the output written to stdout and stderr from the simulation, and the simulation output itself. The simulation output is stored inside a directory named after the basename of the parameter file. An example output directory is below

[mwt@eric2 output-0001]$ ls -al
total 172
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 21:06 .
drwxr-xr-x 4 mwt lsuusers 4096 Sep 27 11:29 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run script from the build configuration into the basedir/<simulation>/SIMFACTORY folder. The executable goes in the exe/ folder, the run and submit scripts into the run/ folder, the Cactus options list into the cfg/ folder, and the parfile into the par/ folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -alR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to The Simulation Factory using the Petashare data storage system. This archiving system can only be used from machines that have access to Petashare. The currently supported machines that have access to Petashare are

queenbee
tezpur
eric
ducky
neptune
zeke
spider
oliver
lacumba
painter
bluedawg
poseidon
louie

In order to use the Petashare archiving system, the Petashare [http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz Pcommands] must be downloaded and installed into your local directory. Once that's done, add a new section in '''etc/defs.local.ini''' for the machine you wish to access petashare from, and add the necessary keys. An example for the machine queenbee is below

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

archivebasepath is where the simulations will be stored on petashare.

Once setup has been completed, you can archive an entire simulation (including all restarts) using the following command

simfactory/sim archive <simulationname>

To archive just a single restart for a given simulation, you can issue the following command

simfactory/sim archive <simulationname> -- restart-id=<restartid>

To get a list of all archived simulations, use the following command

simfactory/sim list-archived-simulations

To retreive an archived simulation, first use the '''list-archived-simulations''' command to retrieve the unique identifier for the simulation, and then issue the following command to retrieve a simulation. It will place the simulation in the current directory.

simfactory/sim get-archived-simulation <archiveid>

=== Macros ===

Simulation Factory Advanced Tutorial

2010-09-30T19:10:26Z

Mthomas: /* Other Advanced Features */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is run for the first time, all the necessary information from the Cactus build configuration is brought into a specific simulation folder created
underneath the '''basedir'''. Contained inside this folder, which has the same name as the specified simulation, are the executable, the run script, the submit script, the SIMFACTORY folder, a log file, and the output directories of each individual restart.

Here is the contents of the simulation folder "btest" with several restarts in it

[mwt@eric2 simulations]$ ls -al btest
total 32
drwxr-xr-x 8 mwt lsuusers 4096 Sep 17 09:03 .
drwxr-xr-x 8 mwt lsuusers 4096 Sep 27 11:32 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0002
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0003
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0004
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0005
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script files needed for submitting and execution, and a properties.ini file that is used by the Simulation Factory to store
information about the simulation.

Each time a simulation is either run or submitted, a restart directory is created underneath the simulation directory. This restart folder has the format of output-####, starting with
output-0001. Contained inside the restart folder are several internal files, the output written to stdout and stderr from the simulation, and the simulation output itself. The simulation output is stored inside a directory named after the basename of the parameter file. An example output directory is below

[mwt@eric2 output-0001]$ ls -al
total 172
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 21:06 .
drwxr-xr-x 4 mwt lsuusers 4096 Sep 27 11:29 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

When a simulation is created, it copies the submit script and the run script from the build configuration into the basedir/<simulation>/SIMFACTORY folder. The executable goes in the exe/ folder, the run and submit scripts into the run/ folder, the Cactus options list into the cfg/ folder, and the parfile into the par/ folder. Below shows an example SIMFACTORY directory

[mwt@eric2 SIMFACTORY]$ ls -alR
.:
total 32
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 cfg
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 data
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:05 exe
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 par
-rw-r--r-- 1 mwt lsuusers 740 Sep 17 09:06 properties.ini
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 run

./cfg:
total 12
-rw-r--r-- 1 mwt lsuusers 4041 Sep 17 09:06 OptionList

./exe:
total 121408
-rwxr-xr-x 1 mwt lsuusers 124306159 Sep 17 09:06 cactus_sim

./par:
total 24
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par

./run:
total 16
-rw-r--r-- 1 mwt lsuusers 1162 Sep 17 09:06 RunScript
-rw-r--r-- 1 mwt lsuusers 410 Sep 17 09:06 SubmitScript

== Other Advanced Features ==

=== Archiving ===

Preliminary support for archiving of simulations has been added to The Simulation Factory using the Petashare data storage system. This archiving system can only be used from machines that have access to Petashare. The currently supported machines that have access to Petashare are

queenbee
tezpur
eric
ducky
neptune
zeke
spider
oliver
lacumba
painter
bluedawg
poseidon
louie

In order to use the Petashare archiving system, the Petashare [http://www.cct.lsu.edu/~sreekanth/petashare/downloads/pcommands/pcommands-2.3.tar.gz Pcommands] must be downloaded and installed into your local directory. Once that's done, add a new section in '''etc/defs.local.ini''' for the machine you wish to access petashare from, and add the necessary keys. An example for the machine queenbee is below

[queenbee]
# archive information
archivetype = petashare
archiveuser = numrel
archivetoolspath = /home/mwt/tools/pcommands-2.3/bin
archivebasepath = /tempZone/home/numrel/mwt/simulations

archivebasepath is where the simulations will be stored on petashare.

Once setup has been completed, you can archive an entire simulation (including all restarts) using the following command

simfactory/sim archive <simulationname>

To archive just a single restart for a given simulation, you can issue the following command

simfactory/sim archive <simulationname> -- restart-id=<restartid>

To get a list of all archived simulations, use the following command

simfactory/sim list-archived-simulations

To retreive an archived simulation, first use the '''list-archived-simulations''' command to retrieve the unique identifier for the simulation, and then issue the following command

simfactory/sim get-archived-simulation <archiveid>

It will place the simulation in the current directory.

=== Macros ===

Simulation Factory Advanced Tutorial

2010-09-30T19:01:36Z

Mthomas: /* Script Locations */

Simulation Factory Advanced Tutorial

2010-09-30T19:01:21Z

Mthomas: /* Script Locations */

Simulation Factory Advanced Tutorial

2010-09-30T19:01:10Z

Mthomas: /* Script Locations */

Simulation Factory Advanced Tutorial

2010-09-30T18:55:40Z

Mthomas: /* What's Produced */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is run for the first time, all the necessary information from the Cactus build configuration is brought into a specific simulation folder created
underneath the '''basedir'''. Contained inside this folder, which has the same name as the specified simulation, are the executable, the run script, the submit script, the SIMFACTORY folder, a log file, and the output directories of each individual restart.

Here is the contents of the simulation folder "btest" with several restarts in it

[mwt@eric2 simulations]$ ls -al btest
total 32
drwxr-xr-x 8 mwt lsuusers 4096 Sep 17 09:03 .
drwxr-xr-x 8 mwt lsuusers 4096 Sep 27 11:32 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0002
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0003
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0004
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0005
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

The SIMFACTORY folder contains the executable, the necessary script files needed for submitting and execution, and a properties.ini file that is used by the Simulation Factory to store
information about the simulation.

Each time a simulation is either run or submitted, a restart directory is created underneath the simulation directory. This restart folder has the format of output-####, starting with
output-0001. Contained inside the restart folder are several internal files, the output written to stdout and stderr from the simulation, and the simulation output itself. The simulation output is stored inside a directory named after the basename of the parameter file. An example output directory is below

[mwt@eric2 output-0001]$ ls -al
total 172
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 21:06 .
drwxr-xr-x 4 mwt lsuusers 4096 Sep 27 11:29 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 17 09:06 LOG
-rw-r--r-- 1 mwt lsuusers 9 Sep 17 09:06 mpd_nodefile
-rw-r--r-- 1 mwt lsuusers 32 Sep 17 09:06 mpi_nodefile
-rw-r--r-- 1 mwt lsuusers 33 Sep 17 09:06 NODELIST
drwxr-xr-x 3 mwt lsuusers 20480 Sep 17 16:12 qc0-mclachlan
-rw------- 1 mwt lsuusers 2520 Sep 17 21:06 qc0-mclachlan.err
-rw------- 1 mwt lsuusers 108210 Sep 17 21:06 qc0-mclachlan.out
-rw-r--r-- 1 mwt lsuusers 13621 Sep 17 09:06 qc0-mclachlan.par
lrwxrwxrwx 1 mwt lsuusers 23 Sep 17 09:06 scratch -> /var/scratch/mwt/250072
drwxr-xr-x 2 mwt lsuusers 4096 Sep 17 09:06 SIMFACTORY

=== Script Locations ===

== Other Advanced Features ==

=== Macros ===
=== Archiving ===

Simulation Factory Advanced Tutorial

2010-09-30T18:50:20Z

Mthomas: /* What's Produced */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===

When a simulation is run for the first time, all the necessary information from the Cactus build configuration is brought into a specific simulation folder created
underneath the '''basedir'''. Contained inside this folder, which has the same name as the specified simulation, are the executable, the run script, the submit script, the SIMFACTORY folder, a log file, and the output directories of each individual restart.

Here is the contents of the simulation folder "btest" with several restarts in it

[mwt@eric2 simulations]$ ls -al btest
total 32
drwxr-xr-x 8 mwt lsuusers 4096 Sep 17 09:03 .
drwxr-xr-x 8 mwt lsuusers 4096 Sep 27 11:32 ..
-rw-r--r-- 1 mwt lsuusers 0 Sep 30 13:30 LOG
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 10:19 output-0001
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:19 output-0002
drwxr-xr-x 4 mwt lsuusers 4096 Aug 20 10:24 output-0003
drwxr-xr-x 3 mwt lsuusers 4096 Aug 20 23:57 output-0004
drwxr-xr-x 4 mwt lsuusers 4096 Sep 17 09:02 output-0005
drwxr-xr-x 7 mwt lsuusers 4096 Aug 20 10:18 SIMFACTORY

When a simulation is either run or submitted, a '''restart''' is created. This restart

=== Script Locations ===

== Other Advanced Features ==

=== Macros ===
=== Archiving ===

Simulation Factory Advanced Tutorial

2010-09-30T18:35:20Z

Mthomas: /* Running a Simulation */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===
=== Script Locations ===

== Other Advanced Features ==

=== Macros ===
=== Archiving ===

Simulation Factory Advanced Tutorial

2010-09-30T18:34:45Z

Mthomas: /* Managing Simulations */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

<li> '''allocation'''
<ul>
<li> The allocation for the simulation to use
<li> '''option''': --allocation
<li> '''default''':
</ul>

<li> '''queue'''
<ul>
<li> The queue for the simulation to use
<li> '''option''': --queue
<li> '''default''': "checkpt"
</ul>
</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, like previously, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options: Running ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===

To launch an interactive session on a compute node, use the following command

simfactory/sim interactive --procs=8 --walltime=4:00:00

To stop a simulation

simfactory/sim stop <simulationname> [--restart-id=<restartid>]

To purge (put in the basedir/TRASH folder) an existing simulation

simfactory/sim purge <simulationname>[--restart-id=<restartid>]

To show the output for a given simulation

simfactory/sim show-output <simulationname> [--restart-id=<restartid>]

=== What's Produced ===
=== Script Locations ===

== Other Advanced Features ==

=== Macros ===
=== Archiving ===

Simulation Factory Advanced Tutorial

2010-09-30T18:29:19Z

Mthomas: /* Managing Simulations */

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central
facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including
LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the
way from creation to output.

== Getting Started ==
In order to begin using The Simulation Factory, it must be checked out from '''svn'''. The Simulation Factory typically resides in the '''simfactory'''
folder inside a Cactus source tree. This can be accomplished with the following '''svn''' command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

== Initial Setup ==

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory
has been checked out into the '''simfactory''' folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit '''simfactory/etc/defs.local.ini''' and replace
<ul>
<li>YOUR_LOGIN with your usual username
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your usual allocation
</ul>

=== Additional Configuration ===

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, '''simfactory/etc/defs.local.ini''' is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine '''queenbee''', you would add the following section:

[queenbee]
user = queenbee_username
allocation = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is '''@USER@'''. This macro expands to the '''user''' property of the Machine Database entry. If user was defined in the '''[default]''' section of '''simfactory/etc/defs.local.ini''' then it will contain that value. An expanded list of useful macros can be found in the [[#Macros]] section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

=== Local Workstation Configuration ===

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local
machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname
can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit '''simfactory/etc/mdb/<hostname>.ini''' and replace

<ul>
<li> '''[generic]''' with '''[<hostname>]'''
<ul>
<li> The section header for this machine database entry must be a unique value and must match the '''nickname''' property exactly.
</ul>
<li> '''nickname = generic''' with '''nickname = <hostname>'''
<li> '''hostname = generic''' with '''hostname = <hostname>'''
<li> '''sourcebasedir = /home/@USER@''' with the correct '''root''' path under which all your Cactus source trees reside.
<li> '''basedir = /home/@USER@/simulations''' with the desired folder for simulation output
</ul>

user, email, and allocation can safely be ignored, as the values from the '''[default]''' section of '''simfactory/etc/defs.local.ini''' will propagate to this entry.

== Accessing Remote Systems ==

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize
an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

=== Information Commands ===

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

=== Syncing ===

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

==== Configuration ====

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

<ul>
<li> '''sourcebasedir'''
<ul>
<li> The root directory under which the Cactus source tree will reside
</ul>
<li> '''basedir'''
<ul>
<li> The root directory which all simulation output will reside
</ul>
<li> '''user'''
<ul>
<li> The username for remote access
</ul>
</ul>

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit '''simfactory/etc/defs.local.ini''' and add an entry for the machine being used. This entry will augment
the existing Machine Database entry, updating the default values with the values specified. An example for the machine '''queenbee''' can be see in the [[#Additional Configuration]] section.

Additionally, to see/modify the list of files and directories that are synchronized, edit '''simfactory/etc/defs.ini''' and find the following two keys

<ul>
<li> '''rsync-sources'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-sourcetree''' is enabled
</ul>
<li> '''rsync-parfiles'''
<ul>
<li> The list of files and directories that will be copied when the option '''--sync-parfiles''' is enabled. This list of files typically includes just parameter files.
</ul>
<li> '''rsync-excludes'''
<ul>
<li> The list of files and directories that will be expressly excluded from syncing
</ul>
</ul>

==== Performing a Sync ====

A sync command takes two arguments, both of which default to '''true'''.

<ul>
<li> '''sync-sourcetree'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-sources''' configuration entry.
</ul>
<li> '''sync-parfiles'''
<ul>
<li> Enable syncing of the list of files and folders specified by the aforementioned '''rsync-parfiles''' configuration entry.
</ul>
</ul>

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

To sync only parfiles, you can negate the '''--sync-sourcetree''' argument with the following command

simfactory/sim sync <machine> --nosync-sourcetree

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

=== Remote Login ===

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

=== Local/Remote Command Execution ===

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/g {} \;" --remotemachine=queenbee

== Cactus Build Configurations ==

The Simulation Factory provides a central facility for configuring and building Cactus source tree. When a Cactus source tree is compiled, The Simulation Factory creates a '''configuration'''
for the compiled executable, storing with it information such as the Cactus options list, and the provided submission and run scripts. This configuration represents the core of what is necessary to perform Cactus execution and submission.

=== Information Commands ===

To list all existing configurations, use the following command

simfactory/sim list-configurations

=== Building a Configuration ===

In order to build a configuration, four pieces of information are needed.

<ul>
<li> Thornlist
<ul>
<li> Default: '''thornlist''' parameter of the Machine Database entry
<li> Override: '''--thornlist=<thornlist>'''
</ul>
<li> Options List
<ul>
<li> Default: '''optionlist''' parameter of the Machine Database entry
<li> Override: '''--optionlist=<optionlist>'''
</ul>
<li> Submission Script
<ul>
<li> Default: '''submitscript''' parameter of the Machine Database entry
<li> Override: '''--submitscript=<submitscript>'''
</ul>
<li> Run Script
<ul>
<li> Default: '''runscript''' parameter of the Machine Database entry
<li> Override: '''--runscript=<runscript>'''
</ul>
</ul>

For any pre-configured Machine Database entry, the defaults for '''optionlist''', '''submitscript''', and '''runscript''' should suffice.

To build a configuration with a specified thornlist, issue the following command:

simfactory/sim build [<configurationname>] --thornlist=<thornlist>

If you choose to omit the configuration name, it will default to 'sim'. If one of the following options is specified, debug, profile, unsafe, and optimise, then the configuration name
will append the specified option onto the end of it. For instance, if you specify --debug with a configuration name 'mybuild', then the configuration name will be mybuild-debug

==== Additional Options ====

<ul>

<li> '''--debug'''
<ul>
<li> Enable debugging in the Cactus executable
</ul>

<li> '''--optimise'''
<ul>
<li> Enable optimisation in the Cactus executable
<li> * WIll be OFF if --debug is enabled.
</ul>

<li> '''--profile'''
<ul>
<li> Build Cactus with profiling
</ul>

<li> '''--unsafe'''
<ul>
<li> Build Cactus with unsafe options
</ul>

<li> '''--reconfig'''
<ul>
<li> Force Cactus to reconfigure before building
</ul>

<li> '''--clean'''
<ul>
<li> Clean Cactus before building
</ul>

</ul>

=== What's Produced ===

The Simulation Factory creates a configuration based upon the input parameters (or defaults) and the compiled executable. Configurations live in the '''configs''' folder inside the Cactus source tree, and compiled executables live inside the '''exe''' folder also inside the Cactus source tree. The following is an example directory structure of the compiled configuration '''sim'''

Cactus/
Cactus/exe/
Cactus/exe/cactus_sim * Follows the naming convention cactus_<configuration>

Cactus/configs/
Cactus/configs/sim/
Cactus/configs/sim/bindings/
Cactus/configs/sim/build/
Cactus/configs/sim/config-data/
Cactus/configs/sim/lib/
Cactus/configs/sim/scratch/
Cactus/configs/sim/OptionList
Cactus/configs/sim/RunScript
Cactus/configs/sim/SubmitScript
Cactus/configs/sim/ThornList

=== Script Locations ===

The Simulation Factory provides default scripts for every one of its preconfigured machines. These scripts can be found in the following locations

<ul>

<li> '''Option Lists'''
<ul>
<li> MDB Key: optionlist
<li> Location: simfactory/etc/optionlists
</ul>

<li> '''Submit Scripts'''
<ul>
<li> MDB Key: submitscript
<li> Location: simfactory/etc/submitscripts
</ul>

<li> '''Run Scripts'''
<ul>
<li> MDB Key: runscript
<li> Location: simfactory/etc/runscripts
</ul>

</ul>

To determine, for instance, which option list queenbee uses by default, issue the following command

simfactory/sim print-mdb queenbee | grep optionlist

== Managing Simulations ==

The Simulation Factory provides a convenient, consistent facility for submitting, executing, and managing simulations. This is accomplished through two main commands: '''submit''' and '''run'''.

=== Information Commands ===

The status of all simulations can be seen with the following command

simfactory/sim list-simulations

If a more detailed look at each simulation is required, the verbose option can be specified

simfactory/sim list-simulations --verbose

=== Submitting a Simulation ===

Four primary pieces of information are necessary when submitting a simulation to the host queuing system. They are

<ul>

<li> '''Configuration'''
<ul>
<li> The Cactus build configuration to use.
<li> '''option''': --configuration
<li> '''default''': "sim"
</ul>

<li> '''Parfile'''
<ul>
<li> The Cactus parameter file to use
<li> '''option''': --parfile
</ul>

<li> '''Walltime'''
<ul>
<li> The amount of CPU time to request
<li> '''option''': --walltime
<li> '''default''': MDB Key: maxwalltime
</ul>

<li> Procs
<ul>
<li> The number of processors to use
<li> '''option''': --procs
<li> '''default''': 1
</ul>

</ul>

'''--configuration''' only needs to be specified the '''first''' time you submit a simulation. Subsequent submissions of the same simulation will use whatever configuration was specified the first time. Here is an example of submitting a simulation named "static_tov" using the aforementioned options

simfactory/sim submit static_tov --configuration sim-debug --parfile=par/static_tov.par --walltime=4:00:00 --procs=8

It is possible to submit a simulation using shorthand, where you can specify the options in a certain order. If you don't specify a simulation name using
the shorthand syntax, it will attempt to derive the simulation name from the basename of the parfile specified.

simfactory/sim submit [<simulationname>] <parfile> <walltime> <procs>

An example

simfactory/sim submit par/static_tov.par 4:00:00 8

==== Additional Options: Submission ====

<ul>

<li> '''Processors Per Node'''
<ul>
<li> The number of processors per node to use.
<li> '''option''': --ppn
<li> '''default''': 1
</ul>

<li> '''Memory'''
<ul>
<li> The amount of memory to use
<li> '''option''': --memory
<li> '''default''': 1024
</ul>

<li> '''cpufreq'''
<ul>
<li> The frequency of the CPU
<li> '''option''': --cpufreq
<li> '''default''': 0
</ul>

</ul>

=== Running a Simulation ===

The Simulation Factory can execute a simulation directly, bypassing a queuing system. Running a simulation directly uses the same options, minus the walltime, as submitting a simulation, only using the '''run''' command instead. An example

simfactory/sim run static_tov --parfile=par/static_tov.par --procs=8

If this simulation does not exist, like previously, '''--configuration=<configuration>''' will need to be specified the first time the simulation is run.

==== Additional Options ====

See [[#Aditional Options: Submission]]

=== Other Simulation Commands ===
=== What's Produced ===
=== Script Locations ===

== Other Advanced Features ==

=== Macros ===
=== Archiving ===

Simulation Factory Advanced Tutorial

2010-09-30T18:19:59Z

Mthomas: /* Managing Simulations */

Simulation Factory Advanced Tutorial

2010-09-30T18:14:10Z

Mthomas: /* Managing Simulations */

Simulation Factory Advanced Tutorial

2010-09-30T18:13:41Z

Mthomas: /* Managing Simulations */

Simulation Factory Advanced Tutorial

2010-09-29T18:42:52Z

Mthomas:

Simulation Factory Advanced Tutorial

2010-09-29T17:55:35Z

Mthomas: /* Managing Simulations */

Simulation Factory Advanced Tutorial

2010-09-29T16:53:12Z

Mthomas: /* Script Locations */

Simulation Factory Advanced Tutorial

2010-09-29T16:50:21Z

Mthomas: /* Script Locations */

Simulation Factory Advanced Tutorial

2010-09-29T16:46:46Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:46:22Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:45:52Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:45:43Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:45:06Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:44:57Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:44:13Z

Mthomas: /* What's Produced */

Simulation Factory Advanced Tutorial

2010-09-29T16:41:29Z

Mthomas:

Simulation Factory Advanced Tutorial

2010-09-29T16:26:30Z

Mthomas: /* = Additional Options */

Simulation Factory Advanced Tutorial

2010-09-29T16:26:21Z

Mthomas: /* Building a Configuration */

Simulation Factory Advanced Tutorial

2010-09-29T16:22:16Z

Mthomas: /* Building a Configuration */

Simulation Factory Advanced Tutorial

2010-09-29T16:19:43Z

Mthomas: /* Building a Configuration */

Simulation Factory Advanced Tutorial

2010-09-29T16:12:29Z

Mthomas: /* Building a Configuration */

Simulation Factory Advanced Tutorial

2010-09-29T16:08:28Z

Mthomas: /* Cactus Build Configurations */

Simulation Factory Advanced Tutorial

2010-09-29T15:57:46Z

Mthomas: /* Local/Remote Command Execution */

Simulation Factory Advanced Tutorial

2010-09-29T15:57:10Z

Mthomas: /* Syncing */

Simulation Factory Advanced Tutorial

2010-09-29T15:54:44Z

Mthomas: /* Syncing */

Simulation Factory Advanced Tutorial

2010-09-29T15:53:41Z

Mthomas: /* Syncing */

Simulation Factory Advanced Tutorial

2010-09-29T15:51:14Z

Mthomas: /* Configuration */

Simulation Factory Advanced Tutorial

2010-09-29T15:48:34Z

Mthomas: /* Performing a Sync */

Simulation Factory Advanced Tutorial

2010-09-29T15:45:52Z

Mthomas: /* Syncing */