Difference between revisions of "Simplified Tutorial for New Users"

From Einstein Toolkit Documentation
Jump to: navigation, search
(add note as to list of packages in option lists)
(add warning about out of date tutorial)
(35 intermediate revisions by 7 users not shown)
Line 1: Line 1:
 +
{|
 +
|-
 +
| [[File:warning.png|128px|frameless|warning]] || This tutorial has not (and will not be) updated for the "Proca" (2019_03) release of the Einstein Toolkit and the instructions provided here will fail if using <nowiki>ET_2019_03</nowiki> instead of <nowiki>ET_2018_09</nowiki>. Please use either the [https://www.einsteintoolkit.org/documentation/new-user-tutorial online tutorial] or the [https://nbviewer.jupyter.org/github/nds-org/jupyter-et/blob/master/CactusTutorial.ipynb offline version of the tutorial notebook] which will be kept up to date. The notebooks contain the same shell commands as used here and they can be copied & pasted into a terminal if desired. || [[File:warning.png|128px|frameless|warning]]
 +
|}
 +
 
Here you will find a step by step guide to downloading, installing, and running the Einstein Toolkit. This tutorial is aimed at Cactus beginners that have never used Cactus before. As such it does not expect any familiarity with Cactus terms and is intended to work on a newly installed workstation or laptop. If you find something that does not work, please feel free to edit the wiki or mail [mailto://users@einsteintoolkit.org users@einsteintoolkit.org]. For a more detailed tutorial, view the [[Tutorial for New Users]].
 
Here you will find a step by step guide to downloading, installing, and running the Einstein Toolkit. This tutorial is aimed at Cactus beginners that have never used Cactus before. As such it does not expect any familiarity with Cactus terms and is intended to work on a newly installed workstation or laptop. If you find something that does not work, please feel free to edit the wiki or mail [mailto://users@einsteintoolkit.org users@einsteintoolkit.org]. For a more detailed tutorial, view the [[Tutorial for New Users]].
  
Line 5: Line 10:
 
You will need a number of packages installed in order to download the Einstein Toolkit components. On a Debian, Ubuntu, Linux-Mint, Fedora or Mac OSX-based system, install them as follows:
 
You will need a number of packages installed in order to download the Einstein Toolkit components. On a Debian, Ubuntu, Linux-Mint, Fedora or Mac OSX-based system, install them as follows:
  
  # Debian (wheezy, jessie)
+
  # Debian (stretch, buster)
  su -c 'apt-get install build-essential libopenmpi-dev openmpi-bin gfortran git subversion curl gnuplot gnuplot-x11'
+
  su -c 'apt-get install build-essential pkg-config libopenmpi-dev openmpi-bin gfortran git subversion curl gnuplot gnuplot-x11'
  # Ubuntu (16.04, 16.10)
+
  # Ubuntu (16.04.2, 17.04, 17.10)
  sudo apt-get install build-essential mpich2 python libmpich2?-dev gfortran git subversion curl gnuplot gnuplot-x11
+
  sudo apt-get install build-essential pkg-config mpich2? python libmpich2?-dev gfortran git subversion curl gnuplot gnuplot-x11
  # Fedora (FC 25)
+
  # Fedora (FC 25, 27)
  su -c ' yum -y install mpich2 python mpich2-devel gsl gsl-devel libjpeg-devel hdf5 hdf5-mpich-devel gcc gcc-c++ gcc-gfortran patch numactl-devel numactl hwloc subversion git openssl-devel lapack-static'
+
  su -c ' yum -y install mpich2 python pkg-config mpich2-devel gsl gsl-devel libjpeg-devel hdf5 hdf5-mpich-devel gcc gcc-c++ gcc-gfortran patch numactl-devel numactl hwloc subversion git openssl-devel lapack-static gnuplot'
  # OSX+MacPorts (Yosemite)
+
  # Mac OS (High Sierra), [https://www.macports.org/index.php MacPorts]
  sudo port install subversion git gnuplot szip jpeg gcc46 fftw fftw-3 gsl openssl hdf5 +fortran +gcc46 -universal zlib openmpi +gcc46
+
  sudo port -N install pkgconfig gcc8 openmpi fftw-3 gsl jpeg zlib hdf5 +fortran +gfortran openssl
  # OSX+Homebrew (Sierra)
+
  # Mac OS (High Sierra), [https://brew.sh/ Homebrew]
brew tap homebrew/science
+
  brew install gnuplot pkg-config gcc fftw gsl hdf5 hwloc jpeg openssl pkg-config szip open-mpi
  brew install subversion gnuplot subversion gcc fftw gsl hdf5 --with-fortran hwloc jpeg openssl pkg-config szip open-mpi
 
  
Installing in addition packages for (Debian package names)
+
For Windows 10, please install the Ubuntu Linux subsystem, then follow the instructions for a Ubuntu system. [https://docs.microsoft.com/en-us/windows/wsl/install-win10 These] are Microsoft's official instructions on how to do so, [https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/ these] are less official but slightly easier to read instructions to do the same. You may also want to install an [https://sourceforge.net/projects/vcxsrv/ X11 server] to provide graphical output.
 +
 
 +
# Windows 10:
 +
https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/
 +
https://sourceforge.net/projects/vcxsrv/
 +
 
 +
=== Speed up installation by using native packages ===
 +
 
 +
Installing in addition packages for (these are Debian package names, so your mileage may vary)
 +
 
 +
''debian/stretch'':
 +
libfftw3-dev libgsl-dev libatlas-base-dev libjpeg-dev libssl-dev libhdf5-serial-dev libhwloc-dev hwloc-nox liblorene-dev libpapi-dev libpfm4-dev
 +
 
 +
''debian/jessie'' (obsolete as of June 2017):
  
 
  libfftw3-dev libgsl0-dev libatlas-base-dev libjpeg-dev libssl-dev libhdf5-serial-dev libhwloc-dev hwloc-nox
 
  libfftw3-dev libgsl0-dev libatlas-base-dev libjpeg-dev libssl-dev libhdf5-serial-dev libhwloc-dev hwloc-nox
Line 29: Line 46:
 
A script called GetComponents is used to fetch the components of the Einstein Toolkit. GetComponents serves as convenient wrapper around lower level tools like git and svn to download the codes that make up the Einstein toolkit from their individual repositories. You may download and make it executable it as follows:
 
A script called GetComponents is used to fetch the components of the Einstein Toolkit. GetComponents serves as convenient wrapper around lower level tools like git and svn to download the codes that make up the Einstein toolkit from their individual repositories. You may download and make it executable it as follows:
  
  curl -O -L https://raw.githubusercontent.com/gridaphobe/CRL/ET_2016_11/GetComponents
+
  curl -O -L https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_09/GetComponents
  chmod a+x GetComponents
+
  chmod u+x GetComponents
  
 
GetComponents accepts a thorn list as an argument. To check out the needed components:
 
GetComponents accepts a thorn list as an argument. To check out the needed components:
  
  ./GetComponents --parallel https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2016_11/einsteintoolkit.th
+
  ./GetComponents https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_09/einsteintoolkit.th
  
 
which downloads the master thorn list from the Einstein toolkit server and proceeds to download all thorns.
 
which downloads the master thorn list from the Einstein toolkit server and proceeds to download all thorns.
Line 46: Line 63:
  
 
  # for Debian
 
  # for Debian
  ./simfactory/bin/sim setup --optionlist=debian.cfg --runscript debian.sh
+
  ./simfactory/bin/sim setup-silent --optionlist=debian.cfg --runscript debian.sh
 
  # for Ubuntu, Mint
 
  # for Ubuntu, Mint
  ./simfactory/bin/sim setup --optionlist=ubuntu.cfg --runscript debian.sh
+
  ./simfactory/bin/sim setup-silent --optionlist=ubuntu.cfg --runscript debian.sh
 
  # for Fedora (you may have to log out and back in if you have just intalled mpich to make the module command work)
 
  # for Fedora (you may have to log out and back in if you have just intalled mpich to make the module command work)
 
  module load mpi
 
  module load mpi
  ./simfactory/bin/sim setup --optionlist=fedora.cfg --runscript debian.sh
+
  ./simfactory/bin/sim setup-silent --optionlist=fedora.cfg --runscript debian.sh
 
 
 
  # OSX+MacPorts
 
  # OSX+MacPorts
  ./simfactory/bin/sim setup --optionlist=osx-macports.cfg --runscript osx-macports.run
+
  ./simfactory/bin/sim setup-silent --optionlist=osx-macports.cfg --runscript osx-macports.run
 
  # OSX+Homebrew
 
  # OSX+Homebrew
export CPATH=/usr/local/include LIBRARY_PATH=/usr/local/lib
+
  ./simfactory/bin/sim setup-silent --optionlist=osx-homebrew.cfg --runscript generic-mpi.run
  ./simfactory/bin/sim setup --optionlist=osx-homebrew.cfg --runscript generic-mpi.run
+
 
 +
After this step is complete you will find your machine's default setup under
 +
./repos/simfactory2/mdb/machines/<hostname >.ini
  
Accept the default values for all options.
+
You can edit some of these settings freely, such as "description", "basedir" etc. Some entry edits could result in simulation start-up warnings and/or errors such as "ppn" (processor-per-node meaning number of cores per cpu), "num-threads" (number of threads per MPI rank) so such edits must be done with some care.
  
 
==Building==
 
==Building==
  
Now that you have configured Simfactory, you may build:
+
Now that you have configured Simfactory, you may build Cactus:
 +
 
 +
./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=manifest/einsteintoolkit.th sim
  
./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=manifest/einsteintoolkit.th
+
which will compile Cactus into an executable <code>exe/cactus_sim</code>
  
 
Adjust <nowiki>-j2</nowiki> to match the number of cores your machine possesses if you want to use more or less than 2 parallel build processes. This may take a while, as it compiles all the thorns specified in manifest/einsteintoolkit.th.
 
Adjust <nowiki>-j2</nowiki> to match the number of cores your machine possesses if you want to use more or less than 2 parallel build processes. This may take a while, as it compiles all the thorns specified in manifest/einsteintoolkit.th.
Line 72: Line 92:
 
The example files provided are mostly too large to run on a single machine. You can try however to run the static_tov example which is smallest and requires about 1.3GB of RAM to run and will run for about 24 hours using a single core. To speed up the run, you can try and reduce the resolution by increasing the parameters "CoordBase::dx", "CoordBase::dy", and "CoordBase::dz" from 8 to 12 which will reduce runtime to roughly 5 hours and memory consumption to 800MB.  
 
The example files provided are mostly too large to run on a single machine. You can try however to run the static_tov example which is smallest and requires about 1.3GB of RAM to run and will run for about 24 hours using a single core. To speed up the run, you can try and reduce the resolution by increasing the parameters "CoordBase::dx", "CoordBase::dy", and "CoordBase::dz" from 8 to 12 which will reduce runtime to roughly 5 hours and memory consumption to 800MB.  
  
   # modify parameter file for smaller memory footprint using sed
+
   # modify parameter file for smaller memory footprint using sed by changing CoordBase::dx from 8 to 12
 
   sed '/CoordBase::d[xyz]/s/8/12/' <par/static_tov.par >par/static_tov_small.par
 
   sed '/CoordBase::d[xyz]/s/8/12/' <par/static_tov.par >par/static_tov_small.par
 
   # start simulation, watch log output
 
   # start simulation, watch log output
   ./simfactory/bin/sim submit static_tov --parfile=par/static_tov_small.par --procs=1 --walltime=8:0:0
+
   ./simfactory/bin/sim submit static_tov --parfile=par/static_tov_small.par --num-threads=1 --procs=1 --walltime=8:0:0
 
   ./simfactory/bin/sim show-output --follow static_tov
 
   ./simfactory/bin/sim show-output --follow static_tov
  
Cactus will use a sinlge MPI process and (unless you have set OMP_NUM_THREADS) will use as many OpenMP threads as you have cores on your computer. Under Linux use can use <code>top -H</code> to show all running threads.
+
Cactus will use a sinlge MPI process and a single OpenMP thread.
  
 
==Look at Results==
 
==Look at Results==
Line 85: Line 105:
  
 
  cd ~/simulations/static_tov/output-0000/static_tov_small
 
  cd ~/simulations/static_tov/output-0000/static_tov_small
 +
ls
  
 
You should see a number of files with the extenstion  ''.asc''. These are 0-D  
 
You should see a number of files with the extenstion  ''.asc''. These are 0-D  
Line 116: Line 137:
 
==Further reading==
 
==Further reading==
 
The following resources provide more in-depth explanations of the tools and concepts used in this tutorial
 
The following resources provide more in-depth explanations of the tools and concepts used in this tutorial
* [http://einsteintoolkit.org/documentation/UsersGuide/UsersGuide.html Cactus users guide] containing detailed information on a thorn structure and the support routines offered by Cactus. Also available in the 'doc' folder of your Cactus checkout in the file 'doc/UsersGuide.pdf'
+
* [https://einsteintoolkit.org/usersguide/UsersGuide.html Cactus users guide] containing detailed information on a thorn structure and the support routines offered by Cactus. Also available in the 'doc' folder of your Cactus checkout in the file 'doc/UsersGuide.pdf'
 +
* [https://github.com/stevenrbrandt/CactusTutorial CactusTutorial] the Cactus tutorial held in 2018 in [https://centra.tecnico.ulisboa.pt/network/grit/einsteintoolkit2018/ Lisbon]
 
* [http://simfactory.org/info/documentation/ Simfactory users guide] which describes the available command for simfactory
 
* [http://simfactory.org/info/documentation/ Simfactory users guide] which describes the available command for simfactory
 +
* This wiki's page on [[Configuring a new machine]]
 
* [https://svn.einsteintoolkit.org/documents/Workshop_Spring_2012/handson_viz/handson_viz.pdf visualization] the visualization lecture of the new user's tutorial held at the APS meeting in Atlanta 2012
 
* [https://svn.einsteintoolkit.org/documents/Workshop_Spring_2012/handson_viz/handson_viz.pdf visualization] the visualization lecture of the new user's tutorial held at the APS meeting in Atlanta 2012
 
* [[ET_Workshop_Spring_2012|course page]] of the new user's tutorial held at the APS meeting in Atlanta 2012
 
* [[ET_Workshop_Spring_2012|course page]] of the new user's tutorial held at the APS meeting in Atlanta 2012
 +
* The [[Main Page#Documentation|Documentation]] section of the wiki

Revision as of 19:39, 31 March 2019

warning This tutorial has not (and will not be) updated for the "Proca" (2019_03) release of the Einstein Toolkit and the instructions provided here will fail if using ET_2019_03 instead of ET_2018_09. Please use either the online tutorial or the offline version of the tutorial notebook which will be kept up to date. The notebooks contain the same shell commands as used here and they can be copied & pasted into a terminal if desired. warning

Here you will find a step by step guide to downloading, installing, and running the Einstein Toolkit. This tutorial is aimed at Cactus beginners that have never used Cactus before. As such it does not expect any familiarity with Cactus terms and is intended to work on a newly installed workstation or laptop. If you find something that does not work, please feel free to edit the wiki or mail users@einsteintoolkit.org. For a more detailed tutorial, view the Tutorial for New Users.

Prerequisites

You will need a number of packages installed in order to download the Einstein Toolkit components. On a Debian, Ubuntu, Linux-Mint, Fedora or Mac OSX-based system, install them as follows:

# Debian (stretch, buster)
su -c 'apt-get install build-essential pkg-config libopenmpi-dev openmpi-bin gfortran git subversion curl gnuplot gnuplot-x11'
# Ubuntu (16.04.2, 17.04, 17.10)
sudo apt-get install build-essential pkg-config mpich2? python libmpich2?-dev gfortran git subversion curl gnuplot gnuplot-x11
# Fedora (FC 25, 27)
su -c ' yum -y install mpich2 python pkg-config mpich2-devel gsl gsl-devel libjpeg-devel hdf5 hdf5-mpich-devel gcc gcc-c++ gcc-gfortran patch numactl-devel numactl hwloc subversion git openssl-devel lapack-static gnuplot'
# Mac OS (High Sierra), MacPorts
sudo port -N install pkgconfig gcc8 openmpi fftw-3 gsl jpeg zlib hdf5 +fortran +gfortran openssl
# Mac OS (High Sierra), Homebrew
brew install gnuplot pkg-config gcc fftw gsl hdf5 hwloc jpeg openssl pkg-config szip open-mpi

For Windows 10, please install the Ubuntu Linux subsystem, then follow the instructions for a Ubuntu system. These are Microsoft's official instructions on how to do so, these are less official but slightly easier to read instructions to do the same. You may also want to install an X11 server to provide graphical output.

# Windows 10:
https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/
https://sourceforge.net/projects/vcxsrv/

Speed up installation by using native packages

Installing in addition packages for (these are Debian package names, so your mileage may vary)

debian/stretch:

libfftw3-dev libgsl-dev libatlas-base-dev libjpeg-dev libssl-dev libhdf5-serial-dev libhwloc-dev hwloc-nox liblorene-dev libpapi-dev libpfm4-dev

debian/jessie (obsolete as of June 2017):

libfftw3-dev libgsl0-dev libatlas-base-dev libjpeg-dev libssl-dev libhdf5-serial-dev libhwloc-dev hwloc-nox

can speed up the compilation, since Cactus will try and use the system provided versions rather than compile its own. Once you have downloaded the Einstein Toolkit, you can inspect the option list files in simfactory/mdb/optionlists/ which often contain a list of recommended packages at the top of the files.

Please make sure all required packages are correctly installed before proceeding with the next step.

Downloading

A script called GetComponents is used to fetch the components of the Einstein Toolkit. GetComponents serves as convenient wrapper around lower level tools like git and svn to download the codes that make up the Einstein toolkit from their individual repositories. You may download and make it executable it as follows:

curl -O -L https://raw.githubusercontent.com/gridaphobe/CRL/ET_2018_09/GetComponents
chmod u+x GetComponents

GetComponents accepts a thorn list as an argument. To check out the needed components:

./GetComponents https://bitbucket.org/einsteintoolkit/manifest/raw/ET_2018_09/einsteintoolkit.th

which downloads the master thorn list from the Einstein toolkit server and proceeds to download all thorns. This thornlist checks out Cactus, the Einstein Toolkit, and Simulation Factory.

Configuring

You may proceed to configure Simfactory which requires some changes for some OS.

cd Cactus
# for Debian
./simfactory/bin/sim setup-silent --optionlist=debian.cfg --runscript debian.sh
# for Ubuntu, Mint
./simfactory/bin/sim setup-silent --optionlist=ubuntu.cfg --runscript debian.sh
# for Fedora (you may have to log out and back in if you have just intalled mpich to make the module command work)
module load mpi
./simfactory/bin/sim setup-silent --optionlist=fedora.cfg --runscript debian.sh
# OSX+MacPorts
./simfactory/bin/sim setup-silent --optionlist=osx-macports.cfg --runscript osx-macports.run
# OSX+Homebrew
./simfactory/bin/sim setup-silent --optionlist=osx-homebrew.cfg --runscript generic-mpi.run

After this step is complete you will find your machine's default setup under ./repos/simfactory2/mdb/machines/<hostname >.ini

You can edit some of these settings freely, such as "description", "basedir" etc. Some entry edits could result in simulation start-up warnings and/or errors such as "ppn" (processor-per-node meaning number of cores per cpu), "num-threads" (number of threads per MPI rank) so such edits must be done with some care.

Building

Now that you have configured Simfactory, you may build Cactus:

./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=manifest/einsteintoolkit.th sim

which will compile Cactus into an executable exe/cactus_sim

Adjust -j2 to match the number of cores your machine possesses if you want to use more or less than 2 parallel build processes. This may take a while, as it compiles all the thorns specified in manifest/einsteintoolkit.th.

Running

The example files provided are mostly too large to run on a single machine. You can try however to run the static_tov example which is smallest and requires about 1.3GB of RAM to run and will run for about 24 hours using a single core. To speed up the run, you can try and reduce the resolution by increasing the parameters "CoordBase::dx", "CoordBase::dy", and "CoordBase::dz" from 8 to 12 which will reduce runtime to roughly 5 hours and memory consumption to 800MB.

 # modify parameter file for smaller memory footprint using sed by changing CoordBase::dx from 8 to 12
 sed '/CoordBase::d[xyz]/s/8/12/' <par/static_tov.par >par/static_tov_small.par
 # start simulation, watch log output
 ./simfactory/bin/sim submit static_tov --parfile=par/static_tov_small.par --num-threads=1 --procs=1 --walltime=8:0:0
 ./simfactory/bin/sim show-output --follow static_tov

Cactus will use a sinlge MPI process and a single OpenMP thread.

Look at Results

When the simulation is complete (ie. when "All done" appears in the output), stop simfactory by pressing CTRL-C and move to the output directory. The command should be:

cd ~/simulations/static_tov/output-0000/static_tov_small
ls

You should see a number of files with the extenstion .asc. These are 0-D (reductions of 3-D grid functions to scalar values) and 1-D ASCII output files that can be plotted with gnuplot.

In this case it's interesting to look at the maximum of the density (in the file hydrobase-rho.maximum.asc). Start gnuplot with the command:

gnuplot

and at the gnuplot prompt type:

plot 'hydrobase-rho.maximum.asc' using 2:3 with linespoints

This plots the data in column 3 (rho) as a function of the data in column 2 (time).

Rho of time lowres.png

As can be seen from the plot, the maximum of the density oscillates with decreasing amplitude around the initial value with a small drift upwards. Even though the initial model is supposed to be in equilibrium, numerical errors means that the numerical model is not exactly in equilibrium and it starts to oscillate. The oscillation energy is slowly dissipated by shocks, decreasing the oscillation amplitude, while the star contracts in response, increasing the maximum density.

A consistent picture can be seen by plotting the minimum of the lapse:

p 'admbase-lapse.minimum.asc' u 2:3 w lp

Lapse of time lowres.png

The quantity shows the same features as the maximum of the density, except the drift is downwards. The downwards trend stems from the contraction of the star. As the star contracts, the curvature of spacetime increases slightly. In response, the singularity avoiding lapse condition used here decreases the lapse.

As the oscillations and subsequent drift of the density and the lapse are caused by numerical error, increasing the numerical resolution will decrease these effects.

Further reading

The following resources provide more in-depth explanations of the tools and concepts used in this tutorial

  • Cactus users guide containing detailed information on a thorn structure and the support routines offered by Cactus. Also available in the 'doc' folder of your Cactus checkout in the file 'doc/UsersGuide.pdf'
  • CactusTutorial the Cactus tutorial held in 2018 in Lisbon
  • Simfactory users guide which describes the available command for simfactory
  • This wiki's page on Configuring a new machine
  • visualization the visualization lecture of the new user's tutorial held at the APS meeting in Atlanta 2012
  • course page of the new user's tutorial held at the APS meeting in Atlanta 2012
  • The Documentation section of the wiki