Pysim Tutorial for New Users

From Einstein Toolkit Documentation
Jump to: navigation, search


This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.

One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.

The example worked through below assumes that you are using the NSF TeraGrid resource Queen Bee. If you don't have an existing account on QueenBee you can request an account to use for these examples.

Download, Configure, and Build

Download GetComponents and make sure it is executable.

chmod a+x GetComponents

Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):

./GetComponents -a													 

This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

Configure the Python Simulation Factory

Before using The Simulation Factory, it is necessary to create a basic configuration for the machine that The Simulation Factory is running on.

To make this easier, The Simulation Factory provides an interactive setup feature.

To start the interactive setup feature, change to the simfactory/bin directory and issue the command

sim setup

Fill in the username, email address, allocation, sourcebasedir, and basedir and then enter N to the question "would you like to enter key/value pairs for a specific machine?".

Compile Executable

./simfactory/bin/sim build --thornlist=manifest/

This may take a few minutes. You may receive an error message that building the utilities failed. This is fine as long as the main executable has been built correctly. You can check this with the command ls exe; if this shows a file cactus_sim, then you are fine.)

Run a Simulation

From the Cactus directory, create a batch queue script and submit it to queue an example simulation:

./simfactory/sim submit static_tov --parfile=par/static_tov.par --procs=32 --walltime=8:0:0

This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.

To check the status of the simulation, use

./simfactory/sim list-simulations

Look at Results

For the following make sure that you have logged in to Queen Bee with X-forwarding enabled, i.e.

ssh -Y

When the simulation is complete, move to the output directory. On Queen Bee ( the command will be (replace <user> with your own login name):

cd /scratch/<user>/simulations/static_tov/output-0001/static_tov

You should see a number of files with the extenstion .asc. These are 0-D (reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase::rho.maximum.asc). Start gnuplot with the command:


and at the gnuplot prompt type:

p 'hydrobase::rho.maximum.asc' u 2:3 w lp

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

rho of time.png

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase::lapse.minimum.asc' u 2:3 w lp

Lapse of time.png

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

Additional Example Parameter Files

In addition to the TOV star parameter file, we also provide a couple of vacuum parameter files: ks-mclachlan.par and qc0-mclachlan.par. The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:

./simfactory/sim create-submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=32 --walltime=8:0:0


./simfactory/sim create-submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=32 --walltime=8:0:0

Additional Simulation Factory Configuration

The example worked out above works out of the box for the Queen Bee machine. You may also be interested though in configuring the Simulation Factory for your own machine or other Teragrid machines. In this section we provide a quick example of what need to be changed or considered in simfactory/etc/udb.ini in order to configure Simulation Factory for your needs. More specifically we are going to work out an example for the TeraGrid machine Ranger.

The first step is to set your username for the machine. This means to modify simfactory/etc/udb.ini customization section to reflect

'YOUR_LOGIN' on Ranger, for example:

   user          = YOUR_LOGIN

You also need to provide your allocation. For example, if your allocation on Ranger is TG-PHY0123456N, underneath the [ranger] entry you just added, enter:

   allocation    = TG-PHY0123456N

Each machine has different policies on which file system to use to build your application and run your simulation or how the file system tree is organized. Simulation Factory keeps a database for several different machines. You can peek at the directory simfactory/etc/mdb to have an idea of the list of machines and what the default configuration is for each of them. In particular for Ranger, the configuration for the source code path and the simulation path defaults to '/work/00507/@USER@' and '/scratch/00507/@USER@/simulations' respectively. You may need to change it to reflect the group number (00507 here) you belong to. Once again, underneath the [ranger] section previously added, set the following keys:

   sourcebasedir = /work/05555/@USER@
   basedir       = /work/05555/@USER@/simulations

After these changes, your entire [ranger] entry will look like the following:

   user          = YOUR_LOGIN
   allocation    = TG-PHY0123456N
   sourcebasedir = /work/05555/@USER@
   basedir       = /work/05555/@USER@/simulations

if your group number is '05555'. With these basic changes to simfactory/etc/udb.ini, you are all set to configure and build the Einstein Toolkit application on Ranger as well.

New machines

Instead of using one of the existing configurations you might want to add a configuration for a new machine. the simfactory/etc/mdb/ directory contains an example for such an entry: generic.ini. To configure this entry, copy the generic.ini file to a new file named after your local machine. For a machine named varia, you would do:

   cp simfactory/etc/mdb/generic.ini simfactory/etc/mdb/varia.ini

where you can replace 'varia' with your machine name. Once this is done, edit this file with your favorite text editor and set various options (nickname, hostname, user, basedir, etc.) for this new machine as above. The generic configuration is supported by a generic optionlist (see simfactory/etc/optionlists/generic.cfg ). These examples require several settings unavailable in that default configuration. You will need to customize an optionlist and set it, either in simfactory/etc/udb.ini as

   optionlist    = optionlistname.cfg

or on the command-line; see the simfactory documentation for more details.

If the machine you added is a bigger resource (e.g. more than your personal laptop or workstation), or can serve as example for some particular machine type, we are happy to include its configuration in the main machine database.

Any Problems?

This example should work smoothly on the TeraGrid machines Queen Bee, Ranger, Kraken and Abe. If you have any problems, please send details to