Configuring a new machine
Contents
Configuring a new machine
If your machine is not supported by SimFactory already, you will need to write your own option list, run script and (for a cluster) submit script.
Machine definition
When using SimFactory on a cluster, it needs to know a lot of information about the details of the cluster, provided in a "machine definition file" in simfactory/mdb/machines. For example, it needs to know the number of cores on each node. Copy one of the provided files, and adapt it to your machine. Getting this right is nontrivial.
When using SimFactory on a laptop or workstation (i.e. a machine for which SimFactory has no machine definition file), the "sim setup" command will write a machine definition file automatically.
Option list
The options provided by Cactus are described in the Cactus documentation. This page provides additional information and recommendations.
The following is based on the ubuntu.cfg optionlist.
VERSION = 2012-09-28
Cactus will reconfigure when the VERSION string changes.
Compilers
CPP = cpp FPP = cpp CC = gcc CXX = g++ F77 = gfortran F90 = gfortran
The C and Fortran preprocessors, and the C, C++, Fortran 77 and Fortran 90 compilers, are specified by these options. You can specify a full path if the compiler you want to use is not available on your default path. Note that it is strongly recommended to use compilers from the same family; e.g. don't mix the Intel C Compiler with the GNU Fortran Compiler.
Compilation and linking flags
CPPFLAGS = -DMPICH_IGNORE_CXX_SEEK FPPFLAGS = -traditional CFLAGS = -g3 -march=native -std=gnu99 CXXFLAGS = -g3 -march=native -std=gnu++0x F77FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none F90FLAGS = -g3 -march=native -fcray-pointer -m128bit-long-double -ffixed-line-length-none LDFLAGS = -rdynamic
Cactus thorns can be written in C or C++. Cactus supports the C99 and C++0x standards respectively. Additionally, the Einstein Toolkit requires the GNU extensions provided by the options gnu99 / gnu++0x. If these extensions are not available, some Einstein Toolkit thorns will not compile.
-g3 ensures that debugging symbols are included in the object files. It is not necessary to set DEBUG = yes to get debugging symbols.
The rdynamic linker flag ensures that additional information is available in the executable for producing backtraces at runtime in the event of an internal error.
LIBDIRS = C_LINE_DIRECTIVES = yes F_LINE_DIRECTIVES = yes
Debugging
DEBUG = no CPP_DEBUG_FLAGS = -DCARPET_DEBUG FPP_DEBUG_FLAGS = -DCARPET_DEBUG C_DEBUG_FLAGS = -O0 CXX_DEBUG_FLAGS = -O0 F77_DEBUG_FLAGS = -O0 F90_DEBUG_FLAGS = -O0
When DEBUG = yes is set (e.g. on the make command line or with SimFactory's --debug option), these debug flags are used. The intention here is to disable optimisation and enable additional code which may slow down execution but makes the code easier to debug.
Optimisation
OPTIMISE = yes CPP_OPTIMISE_FLAGS = -DKRANC_VECTORS # -DCARPET_OPTIMISE -DNDEBUG FPP_OPTIMISE_FLAGS = # -DCARPET_OPTIMISE -DNDEBUG C_OPTIMISE_FLAGS = -O2 -ffast-math CXX_OPTIMISE_FLAGS = -O2 -ffast-math F77_OPTIMISE_FLAGS = -O2 -ffast-math F90_OPTIMISE_FLAGS = -O2 -ffast-math
Profiling
PROFILE = no CPP_PROFILE_FLAGS = FPP_PROFILE_FLAGS = C_PROFILE_FLAGS = -pg CXX_PROFILE_FLAGS = -pg F77_PROFILE_FLAGS = -pg F90_PROFILE_FLAGS = -pg
OpenMP
OPENMP = yes CPP_OPENMP_FLAGS = -fopenmp FPP_OPENMP_FLAGS = -fopenmp C_OPENMP_FLAGS = -fopenmp CXX_OPENMP_FLAGS = -fopenmp F77_OPENMP_FLAGS = -fopenmp F90_OPENMP_FLAGS = -fopenmp
Warnings
WARN = yes CPP_WARN_FLAGS = -Wall FPP_WARN_FLAGS = -Wall C_WARN_FLAGS = -Wall CXX_WARN_FLAGS = -Wall F77_WARN_FLAGS = -Wall F90_WARN_FLAGS = -Wall
ExternalLibraries
The Einstein toolkit thorns use a variety of third-party libraries like MPI or HDF5. These are usually provided by helper thorns in the ExternalLibaries arrangement. As a general rule, to enable a capability FOO add
ExternalLibraries/FOO
to your ThornList and set FOO_DIR to the directory where the include and lib directories are found.
HDF5
If no HDF5 options are given, then HDF5 will be used if it can be automatically detected from standard locations, and will be built from a source package in the HDF5 thorn if not. Alternatively you can specify HDF5_DIR to point to an HF5 installation, for example
HDF5_DIR = /usr/local/hdf5-1.9.1
The following options disable support for Fortran and C++ when building HDF5, as it is not required by the Einstein Toolkit.
HDF5_ENABLE_FORTRAN = no HDF5_ENABLE_CXX = no
MPI
MPI_DIR = /usr MPI_INC_DIRS = /usr/include/mpich2 MPI_LIB_DIRS = /usr/lib MPI_LIBS = mpich fmpich mpl
Others
PTHREADS_DIR = NO_BUILD
Submission script
The submission script is used to submit a job to the queueing system. See the examples in simfactory/mdb/submitscripts, and create a new one for your cluster that uses the same queueing system.
Run script
The most important part of the run script is usually the set of modules that need to be loaded, and the mpirun command to use on the machine. See the examples in simfactory/mdb/runscripts, and create a new one for your cluster that is similar to one that already exists.
