Simulation Factory Advanced Tutorial

The Simulation Factory is an effective method for controlling all facets of a Cactus simulation. It provides a central facility for managing an authoritative source tree, controlling and providing remote access to many commonly-used HPC machines including LONI and the TeraGrid, builds and compiles a Cactus source tree into many independent configurations, and can also manage a simulation all the way from creation to output.

Getting Started

In order to begin using The Simulation Factory, it must be checked out from svn. The Simulation Factory typically resides in the simfactory folder inside a Cactus source tree. This can be accomplished with the following svn command:

svn co https://svn.cct.lsu.edu/repos/numrel/simfactory/branches/PYSIM_2010 simfactory

The Simulation Factory can also be placed in an independent location to be used with multiple Cactus source trees. This approach will be detailed later.

Initial Setup

Once The Simulation Factory has been checked out from svn, the next step is to create two required configuration files. Assuming The Simulation Factory has been checked out into the simfactory folder, this initial configuration can be accomplished with the following commands:

cp simfactory/etc/defs.ini.example simfactory/etc/defs.ini
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini

Edit simfactory/etc/defs.local.ini and replace

• YOUR_LOGIN with your usual username
• YOUR@EMAIL.ADDRESS with your usual email address
• YOUR_ALLOCATION with your usual allocation

Additional Configuration

The Simulation Factory contains a database known as the Machine Database. This collection of information is used to define and help mitigate the uniqueness of each individual HPC machine. The Machine Database is an authoritative collection of information, and is generally not meant to be edited by a user. To add, or change properties of a Machine Database entry, simfactory/etc/defs.local.ini is used. For instance, if an alternative username, allocation, and sourcebasedir is needed for the machine queenbee, you would add the following section:

[queenbee]
user          = queenbee_username
allocation    = queenbee_allocation
sourcebasedir = /work/@USER@

There are several macros that can aide in simplifying configuration. For configuration purposes, the most useful is @USER@. This macro expands to the user property of the Machine Database entry. If user was defined in the [default] section of simfactory/etc/defs.local.ini then it will contain that value. An expanded list of useful macros can be found in the #Macros section

To get a list of preconfigured machines, issue the following command:

simfactory/sim list-machines

Local Workstation Configuration

In order to use a local workstation with The Simulation Factory, a Machine Database entry must be created. Before getting started, the hostname of the local machine must be determined. It is through this hostname that The Simulation Factory matches a Machine Database entry to the executing machine. The hostname can be determined using the following command:

hostname

Once you have the hostname, issue the following command:

cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/<hostname>.ini

Edit simfactory/etc/mdb/<hostname>.ini and replace

• [generic] with [<hostname>]
  • The section header for this machine database entry must be a unique value and must match the nickname property exactly.
• nickname = generic with nickname = <hostname>
• hostname = generic with hostname = <hostname>
• sourcebasedir = /home/@USER@ with the correct root path under which all your Cactus source trees reside.
• basedir = /home/@USER@/simulations with the desired folder for simulation output

user, email, and allocation can safely be ignored, as the values from the [default] section of simfactory/etc/defs.local.ini will propagate to this entry.

Accessing Remote Systems

The Simulation Factory provides a convenient facility for handling remote communication and file transfer with any known machine. Using this facility, a user can synchronize an authoritative source tree, get an interactive shell on the remote system, or execute a command, locally or remotely.

Information Commands

The following commands can be used to discover information about a machine, or list all known, configured machines.

List all known machines

simfactory/sim list-machines

List details about a single machine

simfactory/sim list-machine <machine>

Print the current Machine Database to the screen

simfactory/sim print-mdb

Print the Machine Database entry for a single machine

simfactory/sim print-mdb <machine>

Get the machine that The Simulation Factory is currently being executed on

simfactory/sim print-machine

Syncing

Historically, Cactus and the Einstein Toolkit have not been installed into a central location, and instead are built on-demand for a certain thornlist. In order to aide this approach, The Simulation Factory has the ability to synchronize a Cactus/Einstein Toolkit developer's local, authoritative source tree to a remote HPC machine to be compiled and ran.

Remote access services are implemented on top of ssh, and ssh-like mechanisms such as gsi-ssh. Currently you must manually manage all ssh keys and passwords.

Configuration

Before syncing a small amount of configuration must be performed. It is necessary to either verify the defaults are correct, or to define the correct values for the following keys

• sourcebasedir
  • The root directory under which the Cactus source tree will reside
• basedir
  • The root directory which all simulation output will reside
• user
  • The username for remote access

You can see the configured values by issuing the following command

simfactory/sim print-mdb <machine>

If it is determined that the values for those entries need to be changed. Edit simfactory/etc/defs.local.ini and add an entry for the machine being used. This entry will augment the existing Machine Database entry, updating the default values with the values specified. An example for the machine queenbee can be see in the #Additional Configuration section.

Additionally, to see/modify the list of files and directories that are synchronized, edit simfactory/etc/defs.ini and find the following two keys

• rsync-sources
  • The list of files and directories that will be copied when the option --sync-sourcetree is enabled
• rsync-parfiles
  • The list of files and directories that will be copied when the option --sync-parfiles is enabled. This list of files typically includes just parameter files.
• rsync-excludes
  • The list of files and directories that will be expressly excluded from syncing

Performing a Sync

A sync command takes two arguments, both of which default to true.

• sync-sourcetree
  • Enable syncing of the list of files and folders specified by the aforementioned rsync-sources configuration entry.
• sync-parfiles
  • Enable syncing of the list of files and folders specified by the aforementioned rsync-parfiles configuration entry.

A default sync can be performed by issuing the following command

simfactory/sim sync <machine>

If the desire is to perform a sync from one remote machine to another remote machine, this can be accomplished with the following command

simfactory/sim sync <tomachine> --remotemachine=<frommachine>

Remote Login

The Simulation Factory provides the ability to receive an interactive shell on the remote system. This can be initiated with the following command

simfactory/sim login <machine>

Local/Remote Command Execution

To execute a command locally via The Simulation Factory, use the following command

simfactory/sim execute <command>

If the command is complex, and requires arguments, the command must be quoted. For example

simfactory/sim execute "ls -al"

To execute a remote command, use the following command

simfactory/sim execute <command> --remotemachine=<machine>

An example of a complex command being executed remotely is

simfactory/sim execute "find . -name *.py -exec sed -i .bk -n s/foo/bar/s {} \;" --remotemachine=queenbee