Compiling the Einstein Toolkit

The recommended way to compile the Einstein Toolkit is to use the Simulation Factory ("SimFactory").

Configuring SimFactory for your machine

The ET depends on various libraries, and needs to interact with machine-specific queueing systems and MPI implementations. As such, it needs to be configured for a given machine. For this, it uses SimFactory. Generally, configuring SimFactory means providing an optionlist, for specifying library locations and build options, a submit script for using the batch queueing system, and a runscript, for specifying how Cactus should be run, e.g. which mpirun command to use.

Supported clusters

SimFactory comes with support for many of the large supercomputers. If you are compiling on one of these, it should be sufficient to run

   sim setup

and answer the questions, and SimFactory will automatically detect the correct option list, submit script and run script.

Supported laptop/workstation operating systems

SimFactory also contains general support for specific operating systems commonly used on workstations or laptops, including Mac OS, Ubuntu, Cent OS and Scientific Linux. To configure SimFactory for one of these, you need to find the suitable files in simfactory/mdb/optionlists and simfactory/mdb/runscripts and specify their names on the sim setup command line. The following table lists the files that you need for each supported operating system:

Configuration files

Operating system	Option list	Run script	Notes
Debian	debian.cfg	debian.sh
Ubuntu/Linux Mint	ubuntu.cfg	debian.sh
Fedora	fedora.cfg	debian.sh	Run "module load mpi" before compiling or running, and you may need to log out and back in if you have just installed mpich to make the module command work.
OS X (with MacPorts)	osx-macports.cfg	osx-macports.run
OS X (with HomeBrew)	osx-homebrew.cfg	generic-mpi.run	Run 'export CPATH=/usr/local/include LIBRARY_PATH=/usr/local/lib" before building or running.

You do not need to specify a submit script, as there is no queueing system on a workstation or laptop. Often the optionlist will contain a comment near the top explaining which system packages need to be installed to use the optionlist. For example, for Debian, after installing the packages listed in the comment at the top of debian.cfg,

   sim setup --optionlist=debian.cfg --runscript debian.sh

will configure SimFactory for your machine. For some systems, the runscript can be the general "generic-mpi.sh", or one from a similar OS, for example for Ubuntu you can use debian.sh as a run script.

Unsupported machines

If there is no existing optionlist or runscript (for a cluster, also a submit script) for your machine, you will need to write them yourself. It is recommended to start from one of the already-written files for another machine which is as close as possible to the machine you are trying to use. See the sections below for help with this. Once the files are written, run

   sim setup --optionlist=<optionlist> --runscript <runscript> --submitscript <submitscript>

with the new configuration files. For a cluster, it is unlikely that this will be enough, as SimFactory needs a lot of information about the details of the cluster, provided in a "machine definition file" in simfactory/mdb/machines. Copy an example file, and adapt it to your machine. Getting this right is nontrivial.

Building the Einstein Toolkit

Assuming that SimFactory has been successfully set up on your machine, you should be able to build the Einstein Toolkit with

   sim build --thornlist path/to/thornlist.th

Configuring a new machine

If your machine is not supported by SimFactory already, you will need to write your own option list, run script and (for a cluster) submit script.

Writing your own optionlist

The options provided by Cactus are described in the Cactus documentation. This page provides additional information and recommendations.

The following is based on the ubuntu.cfg optionlist.

VERSION = 2012-09-28

Cactus will reconfigure when the VERSION string changes.

Compilers

CPP = cpp
FPP = cpp
CC  = gcc
CXX = g++
F77 = gfortran
F90 = gfortran

The C and Fortran preprocessors, and the C, C++, Fortran 77 and Fortran 90 compilers, are specified by these options. You can specify a full path if the compiler you want to use is not available on your default path. Note that it is strongly recommended to use compilers from the same family; e.g. don't mix the Intel C Compiler with the GNU Fortran Compiler.

Compilation and linking flags

CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK
FPPFLAGS = -traditional
CFLAGS   = -g3 -march=native -std=gnu99
CXXFLAGS = -g3 -march=native -std=gnu++0x
F77FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none
F90FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none
LDFLAGS  = -rdynamic

Cactus thorns can be written in C or C++. Cactus supports the C99 and C++0x standards respectively. Additionally, the Einstein Toolkit requires the GNU extensions provided by the options gnu99 / gnu++0x. If these extensions are not available, some Einstein Toolkit thorns will not compile.

-g3 ensures that debugging symbols are included in the object files. It is not necessary to set DEBUG = yes to get debugging symbols.

The rdynamic linker flag ensures that additional information is available in the executable for producing backtraces at runtime in the event of an internal error.

LIBDIRS =

C_LINE_DIRECTIVES = yes
F_LINE_DIRECTIVES = yes

Debugging

DEBUG           = no
CPP_DEBUG_FLAGS = -DCARPET_DEBUG
FPP_DEBUG_FLAGS = -DCARPET_DEBUG
C_DEBUG_FLAGS   = -O0 
CXX_DEBUG_FLAGS = -O0 
F77_DEBUG_FLAGS = -O0
F90_DEBUG_FLAGS = -O0

When DEBUG = yes is set (e.g. on the make command line or with SimFactory's --debug option), these debug flags are used. The intention here is to disable optimisation and enable additional code which may slow down execution but makes the code easier to debug.

Optimisation

OPTIMISE           = yes
CPP_OPTIMISE_FLAGS = -DKRANC_VECTORS # -DCARPET_OPTIMISE -DNDEBUG
FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG
C_OPTIMISE_FLAGS   = -O2 -ffast-math
CXX_OPTIMISE_FLAGS = -O2 -ffast-math
F77_OPTIMISE_FLAGS = -O2 -ffast-math
F90_OPTIMISE_FLAGS = -O2 -ffast-math

Profiling

PROFILE           = no
CPP_PROFILE_FLAGS =
FPP_PROFILE_FLAGS =
C_PROFILE_FLAGS   = -pg
CXX_PROFILE_FLAGS = -pg
F77_PROFILE_FLAGS = -pg
F90_PROFILE_FLAGS = -pg

OpenMP

OPENMP           = yes
CPP_OPENMP_FLAGS = -fopenmp
FPP_OPENMP_FLAGS = -fopenmp
C_OPENMP_FLAGS   = -fopenmp
CXX_OPENMP_FLAGS = -fopenmp
F77_OPENMP_FLAGS = -fopenmp
F90_OPENMP_FLAGS = -fopenmp

Warnings

WARN           = yes
CPP_WARN_FLAGS = -Wall
FPP_WARN_FLAGS = -Wall
C_WARN_FLAGS   = -Wall
CXX_WARN_FLAGS = -Wall
F77_WARN_FLAGS = -Wall
F90_WARN_FLAGS = -Wall

ExternalLibraries

The Einstein toolkit thorns use a variety of third-party libraries like MPI or HDF5. These are usually provided by helper thorns in the ExternalLibaries arrangement. As a general rule, to enable a capability FOO add

ExternalLibraries/FOO

to your ThornList and set FOO_DIR to the directory where the include and lib directories are found.

HDF5

If no HDF5 options are given, then HDF5 will be used if it can be automatically detected from standard locations, and will be built from a source package in the HDF5 thorn if not. Alternatively you can specify HDF5_DIR to point to an HF5 installation, for example

HDF5_DIR = /usr/local/hdf5-1.9.1

The following options disable support for Fortran and C++ when building HDF5, as it is not required by the Einstein Toolkit.

HDF5_ENABLE_FORTRAN = no
HDF5_ENABLE_CXX     = no

MPI

MPI_DIR      = /usr
MPI_INC_DIRS = /usr/include/mpich2
MPI_LIB_DIRS = /usr/lib
MPI_LIBS     = mpich fmpich mpl

Others

PTHREADS_DIR = NO_BUILD

Submission script

The submission script is used to submit a job to the queueing system. See the examples in simfactory/mdb/submitscripts, and create a new one for your cluster that uses the same queueing system.

Run script

The most important part of the run script is usually the set of modules that need to be loaded, and the mpirun command to use on the machine. See the examples in simfactory/mdb/runscripts, and create a new one for your cluster that is similar to one that already exists.