Einstein Toolkit Documentation - User contributions [en]

Installing and running on Stampede2

2017-12-22T21:07:13Z

Noncct giedt:

The beginning steps are quite similar to the tutorial for beginners listed above. However, various modifications have to be made in order to accommodate the working environment at Stampede2, which is the successor to Stampede1. The guide given below will walk you through this process. Anyone who knows better ways to do what is outlined here is welcome to edit this page (mercilessly, as stated elsewhere!).

The first step is, as usual, to grab GetComponents and make it executable:

curl -O -L https://raw.githubusercontent.com/gridaphobe/CRL/ET_2017_06/GetComponents
chmod +x GetComponents

Then you have to retrieve everything

./GetComponents https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2017_06/einsteintoolkit.th

Next move into Cactus and run the setup

cd Cactus
./simfactory/bin/sim setup

This will produce the file "simfactory/etc/defs.local.ini". You will be prompted for, among other things, your allocation. That will be something like TG-TRA160063. You will need this in order to run jobs in the batch queues.

Next comes the build. Be prepared to wait quit a while (and leave your computer up, running and connected to Stampede2 for possibly a few hours):

./simfactory/bin/sim build --machine stampede2 --thornlist manifest/einsteintoolkit.th sim-knl

Installing and running on Stampede2

2017-12-17T22:24:08Z

Noncct giedt:

Installing and running on Stampede2

2017-12-17T22:21:35Z

Noncct giedt:

Installing and running on Stampede2

2017-12-17T22:19:11Z

Noncct giedt:

Main Page

2017-12-17T22:16:32Z

Noncct giedt: Installing on Stampede2 is non-trivial! A guide is now under construction.

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==

{{:meeting_agenda}}

== Einstein Toolkit Working Groups and Development Teams ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved. New working groups should be proposed using the [[Working Group Template]].

'''Planned Groups''':

* [[Working List of ET Groups]]

'''Established Groups:'''

* [[Data Dependant Task Scheduler|Data Dependent Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [http://einsteintoolkit.org/documentation/new-user-tutorial Tutorial for New Users]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Installing and running on Stampede2]]
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Tutorial for New Users

2017-12-09T14:53:47Z

Noncct giedt:

This section provides a simple guide for compiling and running an example parameter file for the Einstein Toolkit.

One complexity of high performance computing is the heterogeneous environments presented by different resources with different core software installations and local setups. For example, Einstein Toolkit codes require compiling with MPI and then running the resulting application in a batch queue. This complexity is hidden as much as possible in Cactus and accompanying tools such as the Simulation Factory.

The example worked through below assumes that you are using the LONI resource [http://hpc.loni.org/resources/hpc/system.php?system=QB2]. If you don't have an existing account on Queen Bee you can [http://einsteintoolkit.org/request_testaccount request an account] to use for these examples. The tutorial below assumes that you have some familiarity with Linux/UNIX command-line driven computing. If this is something new to you, there are many resources on-line to provide basic skills in this environment. The terminal environment used below can be opened on any Linux machine, which could be either a base-level installation, or a virtual machine (e.g., using Oracle VirtualBox [https://www.oracle.com/virtualization/virtualbox/index.html]) on top of any platform you are using, including Windows, Mac, etc. Alternatively you can use a terminal client such as Putty [https://www.ssh.com/ssh/putty/].

==Prepare Tools==

Once you have opened a terminal log-in on Queen Bee,
download [https://raw.github.com/gridaphobe/CRL/ET_2017_06/GetComponents GetComponents] to checkout all the components of the Einstein Toolkit, and ensure it is executable:

wget --no-check-certificate https://raw.github.com/gridaphobe/CRL/ET_2017_06/GetComponents
chmod +x GetComponents

==Download, Configure, and Build==

Checkout the Einstein Toolkit using anonymous authentication (this may take a few minutes):

./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2017_06/einsteintoolkit.th

This checks out Cactus, the Einstein Toolkit thorns, the Simulation Factory and example parameter files into a directory named Cactus.

Note: Some versions of svn might show problems with the parallel checkout. If you see errors like (svn: E155037: Previous operation has not finished), try without the --parallel option.

Note: If svn checkout fails for certain repository domains with a message like "Server SSL certificate verification failed: issuer is not trusted", you can force your system to accept the new certificate by checking out one of that repository's modules by hand. E.g.,

svn co http://svn.cactuscode.org/projects/ExternalLibraries/pciutils/branches/ET_2017_06

This will fail with a similar message to before, but will give you the option of accepting the certificate. Accept it permanently (p), and hit Enter. Running the GetComponents script should now work for all modules on that repository.

===Configure the Simulation Factory===


mkdir /scratch/$USER/simulations
cd Cactus
cp simfactory/etc/defs.local.ini.simple simfactory/etc/defs.local.ini
nano simfactory/etc/defs.local.ini

Edit defs.local.ini and replace
<ul>
<li>YOUR_LOGIN with your username on Queen Bee
<li>YOUR@EMAIL.ADDRESS with your usual email address
<li>YOUR_THORNLIST with <tt>thornlists/einsteintoolkit.th</tt>
</ul>

Uncomment the [qb] section in defs.local.ini and replace
<ul>
<li>YOUR_QUEENBEE_LOGIN with your username on Queen Bee
<li>YOUR_ALLOCATION with your project allocation on Queen Bee. Type 'balance' on the command line and choose one of the project names. It is assumed
that you got a small start-up allocation with your account. Please contact Frank Löffler <knarf@cct.lsu.edu> if you didn't.
</ul>

===Compile Executable===

./simfactory/bin/sim build

This will take quite a few minutes. It may also take longer, depending on the load on the machine (especially the number of used compiler licenses). This is a good time to think about a cup of tea or coffee. Last time I tried it took 20 minutes. The executable can be found as exe/cactus_sim. Don't run this yet, however, because the Queen Bee head node (the computer you are logged into) is a shared resource: compute jobs are not supposed to be run directly there. For this, create a "batch job" (read on).

==Run a Simulation==

If you run for the first time on Queen Bee, create the directory that will later hold the simulation results:

mkdir /scratch/$USER/simulations

From the Cactus directory, create a batch queue script and submit it to queue an example simulation:

./simfactory/bin/sim submit static_tov --parfile=par/static_tov.par --procs=40 --walltime=8:0:0

This parameter file sets up a static TOV star (a model of a single neutron star) with a mass of 1.4 solar masses and integrates the combined relativistic fluid dynamics and spacetime evolution equations in time. The spacetime is evolved using the BSSN 3+1 formulation of Einstein's equations and the fluid is evolved using a high resolution shock capturing method.

To check the status of the simulation, use

./simfactory/bin/sim list-simulations

==Look at Results==

For the following make sure that you have logged in to Queen Bee with X-forwarding enabled, i.e.

ssh -Y qb.loni.org

When the simulation is complete, move to the output directory. On Queen Bee
(qb.loni.org) the command will be:

cd /scratch/$USER/simulations/static_tov/output-0000/static_tov

You should see a number of files with the extenstion ''.asc''. These are 0-D
(reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that
can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase::rho.maximum.asc). Start gnuplot with the command:

gnuplot

and at the gnuplot prompt type:

p 'hydrobase-rho.maximum.asc' u 2:3 w lp

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

[[Image:rho_of_time.png]]

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase-lapse.minimum.asc' u 2:3 w lp

[[Image:Lapse_of_time.png]]

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

==Additional Example Parameter Files==

In addition to the TOV star parameter file, we also provide a couple of vacuum parameter files: ks-mclachlan.par and qc0-mclachlan.par.
The first (ks-mclachlan.par) evolves a single isolated rotating black hole with the initial data (M=1 and a/M=0.8) specified in Kerr-Schild coordinates. The second (qc0-mclachlan.par) evolves a binary black hole system using the moving puncture technique. The black holes start at a close separation and only complete about one half of an orbit before merging. You can run these additional parameter files using a similar command in the Cactus directory as for the TOV star above:

./simfactory/bin/sim submit ks-mclachlan --parfile=par/ks-mclachlan.par --procs=40 --walltime=8:0:0

and

./simfactory/bin/sim submit qc0-mclachlan --parfile=par/qc0-mclachlan.par --procs=40 --walltime=8:0:0

==Additional Simulation Factory Configuration==

The example worked out above works out of the box for the Queen Bee machine.
You may also be interested though in configuring the Simulation Factory for your
own machine or other Xsede machines. In this section we provide a quick
example of what needs to be changed or considered in simfactory/etc/defs.local.ini
in order to configure Simulation Factory for your needs. More specifically
we are going to work out an example for the XSEDE machine Stampede.

Simfactory already knows about Stampede, so we only need to provide the settings specific to your account. To do this, you create a section for Stampede machine in defs.local.ini:

[stampede]
user = YOUR_STAMPEDE_USERNAME
allocation = YOUR_STAMPEDE_ALLOCATION

Each machine has different policies on which file system to use to build
your application and run your simulation or how the file system tree
is organized. Simulation Factory keeps a database for several different
machines. You can peek at simfactory/mdb/machines to have an idea of the list
of machines and what the default configuration is for each of them.
In particular for Stampede, the configuration for the source code path and
the simulation path defaults to '/work/00507/@USER@' and
'/scratch/00507/@USER@/simulations' respectively. You may need to
change it to reflect the group number (00507 here) you belong to.
For example you could set (in the [stampede] section):

sourcebasedir = /work/05555/@USER@
basedir = /work/05555/@USER@/simulations

if your group number is '05555'. With these basic changes to
simfactory/etc/defs.local.ini you are all set to configure and build the
Einstein Toolkit application on Stampede as well.

==Going further==

===Adding a new machine to Simfactory===

This and other topics are explained in the [[Simulation_Factory_Advanced_Tutorial]].

===Development version of the Einstein Toolkit===

Instead of using the release version listed above, you might want the latest development version. Proceed as above, but download from the trunk:
(GetComponents from trunk)
.... https://bitbucket.org/einsteintoolkit/manifest/raw/master/einsteintoolkit.th

===More about Cactus===
The EinsteinToolkit is built on top of Cactus. Development requires some familiarity with it. Start reading
* [https://svn.cactuscode.org/tutorials/introduction/introduction.pdf Introduction to cactus]
* [http://cactuscode.org/documentation/tutorials/ Cactus tutorials], particularly the [http://cactuscode.org/documentation/tutorials/wavetoydemo/ WaveToy demo]

==Any Problems?==

This example should work smoothly on Queen Bee. If you have any problems, please send details to [mailto:users@einsteintoolkit.org users@einsteintoolkit.org].

User:Noncct giedt

2017-12-07T19:05:31Z

Noncct giedt: Info about Joel Giedt

Joel Giedt
Associate Professor
Department of Physics, Applied Physics and Astronomy
Theoretical particle physicist with interests in quantum gravity, electroweak and QCD physics in the extreme environments of collapsing stars.