Difference between revisions of "Tutorial for New Users"

From Einstein Toolkit Documentation
Jump to: navigation, search
(Remove --noshallow and -a options from GetComponents call; they are now the default)
(add warning about tutorial beng obsolete)
(40 intermediate revisions by 17 users not shown)
Line 1: Line 1:
 +
'''This tutorial is obsolete.''' Please use: the [http://einsteintoolkit.org/documentation/new-user-tutorial jupyter based one instead]
 +
 
This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.
 
This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.
  
 
One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.
 
One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.
  
The example worked through below assumes that you are using the NSF TeraGrid resource [http://www.loni.org/systems/system.php?system=QueenBee Queen Bee]. If you don't have an existing account on Queen Bee you can [http://einsteintoolkit.org/documentation/request request an account] to use for these examples.
+
The example worked through below assumes that you are using the LONI resource [http://hpc.loni.org/resources/hpc/system.php?system=QB2]. If you don't have an existing account on Queen Bee you can [http://einsteintoolkit.org/request_testaccount request an account] to use for these examples.  The tutorial below assumes that you have some familiarity with Linux/UNIX command-line driven computing.  If this is something new to you, there are many resources on-line to provide basic skills in this environment.  The terminal environment used below can be opened on any Linux machine, which could be either a base-level installation, or a virtual machine (e.g., using Oracle VirtualBox [https://www.oracle.com/virtualization/virtualbox/index.html]) on top of any platform you are using, including Windows, Mac, etc.  Alternatively you can use a terminal client such as Putty [https://www.ssh.com/ssh/putty/].
  
 
==Prepare Tools==
 
==Prepare Tools==
  
On Queen Bee (qb.loni.org), the <tt>git</tt> and <tt>hg</tt> clients may not be in your path by default. To add them type:
+
Once you have opened a terminal log-in on Queen Bee,  
 
+
download [https://raw.github.com/gridaphobe/CRL/ET_2017_06/GetComponents GetComponents] to checkout all the components of the Einstein Toolkit, and ensure it is executable:
soft add +python-2.6.4-gcc-4.3.2
 
soft add +git
 
export PATH=$PATH:/home/eschnett/mercurial-1.9.3/bin
 
 
 
Download [https://github.com/gridaphobe/CRL/raw/ET_2012_05/GetComponents GetComponents] to checkout all the components of the Einstein Toolkit, and ensure it is executable:
 
  
  wget --no-check-certificate https://github.com/gridaphobe/CRL/raw/ET_2012_05/GetComponents  
+
  wget --no-check-certificate https://raw.github.com/gridaphobe/CRL/ET_2017_06/GetComponents  
  chmod 755 GetComponents
+
  chmod +x GetComponents
  
 
==Download, Configure, and Build==
 
==Download, Configure, and Build==
Line 22: Line 19:
 
Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):
 
Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):
  
  ./GetComponents http://svn.einsteintoolkit.org/manifest/branches/ET_2012_05/einsteintoolkit.th
+
  ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2017_06/einsteintoolkit.th
  
 
This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.
 
This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.
 +
 +
Note: Some versions of svn might show problems with the parallel checkout. If you see errors like (svn: E155037: Previous operation has not finished), try without the --parallel option.
 +
 +
Note: If svn checkout fails for certain repository domains with a message like "Server SSL certificate verification failed: issuer is not trusted", you can force your system to accept the new certificate by checking out one of that repository's modules by hand. E.g.,
 +
 +
svn co http://svn.cactuscode.org/projects/ExternalLibraries/pciutils/branches/ET_2017_06
 +
 +
This will fail with a similar message to before, but will give you the option of accepting the certificate. Accept it permanently (p), and hit Enter. Running the GetComponents script should now work for all modules on that repository.
  
 
===Configure the Simulation Factory===
 
===Configure the Simulation Factory===
Line 30: Line 35:
 
<!--mkdir below is a workaround until simfactory creates its own simulation
 
<!--mkdir below is a workaround until simfactory creates its own simulation
 
directory (rhaas, 2011-10-25)-->
 
directory (rhaas, 2011-10-25)-->
  mkdir scratch/simulations
+
  mkdir /scratch/$USER/simulations
 
  cd Cactus
 
  cd Cactus
 
  cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
 
  cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
Line 39: Line 44:
 
<li>YOUR_LOGIN with your username on Queen Bee
 
<li>YOUR_LOGIN with your username on Queen Bee
 
<li>YOUR@EMAIL.ADDRESS with your usual email address
 
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_ALLOCATION with your project allocation on Queen Bee
+
<li>YOUR_THORNLIST with <tt>thornlists/einsteintoolkit.th</tt>
<li>YOUR_THORNLIST with <tt>manifest/einsteintoolkit.th</tt>
+
</ul>
 +
 
 +
Uncomment the [qb] section in defs.local.ini and replace
 +
<ul>
 +
<li>YOUR_QUEENBEE_LOGIN with your username on Queen Bee
 +
<li>YOUR_ALLOCATION with your project allocation on Queen Bee. Type 'balance' on the command line and choose one of the project names. It is assumed
 +
that you got a small start-up allocation with your account. Please contact Frank Löffler <knarf@cct.lsu.edu> if you didn't.
 
</ul>
 
</ul>
  
Line 47: Line 58:
 
  ./simfactory/bin/sim build
 
  ./simfactory/bin/sim build
  
This may take a few minutes. You may receive an error message that building the utilities
+
This will take quite a few minutes. It may also take longer, depending on the load on the machine (especially the number of used compiler licenses). This is a good time to think about a cup of tea or coffee. Last time I tried it took 20 minutes. The executable can be found as exe/cactus_sim. Don't run this yet, however, because the Queen Bee head node (the computer you are logged into) is a shared resource: compute jobs are not supposed to be run directly there. For this, create a "batch job" (read on).
failed. This is fine as long as the main executable has been built correctly.  You can check this with the command <tt>ls exe</tt>; if this shows a file <tt>cactus_sim</tt>, then you are fine.)
 
  
 
==Run a Simulation==
 
==Run a Simulation==
 +
 +
If you run for the first time on Queen Bee, create the directory that will later hold the simulation results:
 +
 +
mkdir /scratch/$USER/simulations
  
 
From the Cactus directory, create a batch queue script and submit it to queue an example simulation:
 
From the Cactus directory, create a batch queue script and submit it to queue an example simulation:
+
 
  ./simfactory/bin/sim submit static_tov --parfile=par/static_tov.par --procs=32 --walltime=8:0:0
+
  ./simfactory/bin/sim submit static_tov --parfile=par/static_tov.par --procs=40 --walltime=8:0:0
  
 
This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.
 
This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.
Line 71: Line 85:
 
(qb.loni.org) the command will be:
 
(qb.loni.org) the command will be:
  
  cd ~/scratch/simulations/static_tov/output-0000/static_tov
+
  cd /scratch/$USER/simulations/static_tov/output-0000/static_tov
  
 
You should see a number of files with the extenstion  ''.asc''. These are 0-D  
 
You should see a number of files with the extenstion  ''.asc''. These are 0-D  
Line 83: Line 97:
 
and at the gnuplot prompt type:
 
and at the gnuplot prompt type:
  
  p 'hydrobase::rho.maximum.asc' u 2:3 w lp
+
  p 'hydrobase-rho.maximum.asc' u 2:3 w lp
  
 
This plots the data in column 3 (rho) as a function of the data in column 2 (time).
 
This plots the data in column 3 (rho) as a function of the data in column 2 (time).
Line 93: Line 107:
 
A consistent picture can be seen by plotting the minimum of the lapse:
 
A consistent picture can be seen by plotting the minimum of the lapse:
  
  p 'admbase::lapse.minimum.asc' u 2:3 w lp
+
  p 'admbase-lapse.minimum.asc' u 2:3 w lp
  
 
[[Image:Lapse_of_time.png]]
 
[[Image:Lapse_of_time.png]]
Line 106: Line 120:
 
The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:
 
The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:
  
  ./simfactory/bin/sim submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=32 --walltime=8:0:0
+
  ./simfactory/bin/sim submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=40 --walltime=8:0:0
  
 
and
 
and
  
  ./simfactory/bin/sim submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=32 --walltime=8:0:0
+
  ./simfactory/bin/sim submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=40 --walltime=8:0:0
  
 
==Additional Simulation Factory Configuration==
 
==Additional Simulation Factory Configuration==
Line 116: Line 130:
 
The example worked out above works out of the box for the Queen Bee machine.
 
The example worked out above works out of the box for the Queen Bee machine.
 
You may also be interested though in configuring the Simulation Factory for your
 
You may also be interested though in configuring the Simulation Factory for your
own machine or other Teragrid machines. In this section we provide a quick
+
own machine or other Xsede machines. In this section we provide a quick
 
example of what needs to be changed or considered in simfactory/etc/defs.local.ini
 
example of what needs to be changed or considered in simfactory/etc/defs.local.ini
 
in order to configure Simulation Factory for your needs. More specifically
 
in order to configure Simulation Factory for your needs. More specifically
we are going to work out an example for the TeraGrid machine Ranger.
+
we are going to work out an example for the XSEDE machine Stampede.
  
Simfactory already knows about Ranger, so we only need to provide the settings specific to your account. To do this, you create a section for Ranger machine in defs.local.ini:
+
Simfactory already knows about Stampede, so we only need to provide the settings specific to your account. To do this, you create a section for Stampede machine in defs.local.ini:
  
  [ranger]
+
  [stampede]
  user = YOUR_RANGER_USERNAME
+
  user = YOUR_STAMPEDE_USERNAME
  allocation = YOUR_RANGER_ALLOCATION
+
  allocation = YOUR_STAMPEDE_ALLOCATION
  
 
Each machine has different policies on which file system to use to build
 
Each machine has different policies on which file system to use to build
Line 132: Line 146:
 
machines. You can peek at simfactory/mdb/machines to have an idea of the list
 
machines. You can peek at simfactory/mdb/machines to have an idea of the list
 
of machines and what the default configuration is for each of them.
 
of machines and what the default configuration is for each of them.
In particular for Ranger, the configuration for the source code path and
+
In particular for Stampede, the configuration for the source code path and
 
the simulation path defaults to '/work/00507/@USER@' and
 
the simulation path defaults to '/work/00507/@USER@' and
 
'/scratch/00507/@USER@/simulations' respectively. You may need to
 
'/scratch/00507/@USER@/simulations' respectively. You may need to
 
change it to reflect the group number (00507 here) you belong to.
 
change it to reflect the group number (00507 here) you belong to.
For example you could set (in the [ranger] section):
+
For example you could set (in the [stampede] section):
  
 
  sourcebasedir = /work/05555/@USER@
 
  sourcebasedir = /work/05555/@USER@
Line 143: Line 157:
 
if your group number is '05555'. With these basic changes to
 
if your group number is '05555'. With these basic changes to
 
simfactory/etc/defs.local.ini you are all set to configure and build the
 
simfactory/etc/defs.local.ini you are all set to configure and build the
Einstein Toolkit application on Ranger as well.
+
Einstein Toolkit application on Stampede as well.
  
 
==Going further==
 
==Going further==
Line 155: Line 169:
 
Instead of using the release version listed above, you might want the latest development version.  Proceed as above, but download from the trunk:
 
Instead of using the release version listed above, you might want the latest development version.  Proceed as above, but download from the trunk:
 
   (GetComponents from trunk)
 
   (GetComponents from trunk)
   .... -a http://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
+
   .... https://bitbucket.org/einsteintoolkit/manifest/raw/master/einsteintoolkit.th
  
 
===More about Cactus===
 
===More about Cactus===
Line 164: Line 178:
 
==Any Problems?==
 
==Any Problems?==
  
This example should work smoothly on the TeraGrid machines Queen Bee, Ranger, Kraken and Abe. If you have any problems, please send details to [mailto:users@einsteintoolkit.org users@einsteintoolkit.org].
+
This example should work smoothly on Queen Bee. If you have any problems, please send details to [mailto:users@einsteintoolkit.org users@einsteintoolkit.org].

Revision as of 22:54, 12 December 2017

This tutorial is obsolete. Please use: the jupyter based one instead

This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.

One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.

The example worked through below assumes that you are using the LONI resource [1]. If you don't have an existing account on Queen Bee you can request an account to use for these examples. The tutorial below assumes that you have some familiarity with Linux/UNIX command-line driven computing. If this is something new to you, there are many resources on-line to provide basic skills in this environment. The terminal environment used below can be opened on any Linux machine, which could be either a base-level installation, or a virtual machine (e.g., using Oracle VirtualBox [2]) on top of any platform you are using, including Windows, Mac, etc. Alternatively you can use a terminal client such as Putty [3].

Prepare Tools

Once you have opened a terminal log-in on Queen Bee, download GetComponents to checkout all the components of the Einstein Toolkit, and ensure it is executable:

wget --no-check-certificate https://raw.github.com/gridaphobe/CRL/ET_2017_06/GetComponents 
chmod +x GetComponents

Download, Configure, and Build

Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):

./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2017_06/einsteintoolkit.th

This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

Note: Some versions of svn might show problems with the parallel checkout. If you see errors like (svn: E155037: Previous operation has not finished), try without the --parallel option.

Note: If svn checkout fails for certain repository domains with a message like "Server SSL certificate verification failed: issuer is not trusted", you can force your system to accept the new certificate by checking out one of that repository's modules by hand. E.g.,

svn co http://svn.cactuscode.org/projects/ExternalLibraries/pciutils/branches/ET_2017_06

This will fail with a similar message to before, but will give you the option of accepting the certificate. Accept it permanently (p), and hit Enter. Running the GetComponents script should now work for all modules on that repository.

Configure the Simulation Factory

mkdir /scratch/$USER/simulations
cd Cactus
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
nano simfactory/etc/defs.local.ini

Edit defs.local.ini and replace

  • YOUR_LOGIN with your username on Queen Bee
  • YOUR@EMAIL.ADDRESS with your usual email address
  • YOUR_THORNLIST with thornlists/einsteintoolkit.th

Uncomment the [qb] section in defs.local.ini and replace

  • YOUR_QUEENBEE_LOGIN with your username on Queen Bee
  • YOUR_ALLOCATION with your project allocation on Queen Bee. Type 'balance' on the command line and choose one of the project names. It is assumed that you got a small start-up allocation with your account. Please contact Frank Löffler <knarf@cct.lsu.edu> if you didn't.

Compile Executable

./simfactory/bin/sim build

This will take quite a few minutes. It may also take longer, depending on the load on the machine (especially the number of used compiler licenses). This is a good time to think about a cup of tea or coffee. Last time I tried it took 20 minutes. The executable can be found as exe/cactus_sim. Don't run this yet, however, because the Queen Bee head node (the computer you are logged into) is a shared resource: compute jobs are not supposed to be run directly there. For this, create a "batch job" (read on).

Run a Simulation

If you run for the first time on Queen Bee, create the directory that will later hold the simulation results:

mkdir /scratch/$USER/simulations

From the Cactus directory, create a batch queue script and submit it to queue an example simulation:

./simfactory/bin/sim submit static_tov --parfile=par/static_tov.par --procs=40 --walltime=8:0:0

This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.

To check the status of the simulation, use

./simfactory/bin/sim list-simulations

Look at Results

For the following make sure that you have logged in to Queen Bee with X-forwarding enabled, i.e.

ssh -Y qb.loni.org

When the simulation is complete, move to the output directory. On Queen Bee (qb.loni.org) the command will be:

cd /scratch/$USER/simulations/static_tov/output-0000/static_tov

You should see a number of files with the extenstion .asc. These are 0-D (reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase::rho.maximum.asc). Start gnuplot with the command:

gnuplot

and at the gnuplot prompt type:

p 'hydrobase-rho.maximum.asc' u 2:3 w lp

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

rho of time.png

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase-lapse.minimum.asc' u 2:3 w lp

Lapse of time.png

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

Additional Example Parameter Files

In addition to the TOV star parameter file, we also provide a couple of vacuum parameter files: ks-mclachlan.par and qc0-mclachlan.par. The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:

./simfactory/bin/sim submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=40 --walltime=8:0:0

and

./simfactory/bin/sim submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=40 --walltime=8:0:0

Additional Simulation Factory Configuration

The example worked out above works out of the box for the Queen Bee machine. You may also be interested though in configuring the Simulation Factory for your own machine or other Xsede machines. In this section we provide a quick example of what needs to be changed or considered in simfactory/etc/defs.local.ini in order to configure Simulation Factory for your needs. More specifically we are going to work out an example for the XSEDE machine Stampede.

Simfactory already knows about Stampede, so we only need to provide the settings specific to your account. To do this, you create a section for Stampede machine in defs.local.ini:

[stampede]
user = YOUR_STAMPEDE_USERNAME
allocation = YOUR_STAMPEDE_ALLOCATION

Each machine has different policies on which file system to use to build your application and run your simulation or how the file system tree is organized. Simulation Factory keeps a database for several different machines. You can peek at simfactory/mdb/machines to have an idea of the list of machines and what the default configuration is for each of them. In particular for Stampede, the configuration for the source code path and the simulation path defaults to '/work/00507/@USER@' and '/scratch/00507/@USER@/simulations' respectively. You may need to change it to reflect the group number (00507 here) you belong to. For example you could set (in the [stampede] section):

sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

if your group number is '05555'. With these basic changes to simfactory/etc/defs.local.ini you are all set to configure and build the Einstein Toolkit application on Stampede as well.

Going further

Adding a new machine to Simfactory

This and other topics are explained in the Simulation_Factory_Advanced_Tutorial.

Development version of the Einstein Toolkit

Instead of using the release version listed above, you might want the latest development version. Proceed as above, but download from the trunk:

 (GetComponents from trunk)
 .... https://bitbucket.org/einsteintoolkit/manifest/raw/master/einsteintoolkit.th

More about Cactus

The EinsteinToolkit is built on top of Cactus. Development requires some familiarity with it. Start reading

Any Problems?

This example should work smoothly on Queen Bee. If you have any problems, please send details to users@einsteintoolkit.org.