Testsuite Machines

This page contains notes and instructions for people running the ET testsuites on various different machines. If you have experience running testsuites on a machine which is not listed here, please consider adding some information which might help others (or yourself!) in future.

General

To check out the ET:

 mkdir etrelease
 cd etrelease
 curl -O https://github.com/gridaphobe/CRL/raw/master/GetComponents 
 chmod a+x GetComponents
 ./GetComponents --root=. -a https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th

 cp udb.example.pm udb.pm

 Replace "redshift" with the name of your local machine, unless you are running on one of the supported machines.

 Replace YOUR_LOGIN, YOUR_EMAIL_ADDRESS and YOUR_ALLOCATION as appropriate.

 Set sourcebasedir for your local machine to be the directory containing etrelease (do not set it to etrelease itself).

 Set your username on each machine:

   set_option 'MACHINE'  , 'user', 'USERNAME';

Machines

Kraken

Edit simfactory/mdb.pm

 -        myproxy-logon -p 7514 -s myproxy.teragrid.org -T -l @USER@ -o @SOURCEDIR@/.globus/proxy-teragrid
 +        myproxy-logon -p 7514 -s myproxy.teragrid.org -T -l <username> -o @SOURCEDIR@/.globus/proxy-teragrid

where <username> is your user name on Kraken. This is related to Ticket #381

Change the sourcebasedir in simfactory/udb.pm to be under /lustre/scratch. This is necessary because the Cactus directory must be visible from the compute node when running the tests (to see the parameter files, testsuite reference output and test.ccl files).

   set_option 'kraken', 'sourcebasedir', '/lustre/scratch/USERNAME';

Copy kraken.sh as kraken-testsuite.sh in simfactory/scriptfiles. Replace the aprun command with the following:

cd @SOURCEDIR@

export CCTK_TESTSUITE_RUN_COMMAND="aprun -n \$nprocs -d 1 \$exe \$parfile"

test_dir=TEST_$(date +%Y-%m-%d-%H%M%S) mkdir $test_dir

CONFIGNAME=$(ls configs|tail) # SimFactory should provide the configuration name

for i in 1 2 3; do

 case $i in
   1 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=1
     export OMP_NUM_THREADS=1;;
   2 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=2
     export OMP_NUM_THREADS=1;;
   3 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=2
     export OMP_NUM_THREADS=2;;
 esac
 make $CONFIGNAME-testsuite PROMPT=no
 cp TEST/$CONFIGNAME/summary.log kraken__${CCTK_TESTSUITE_RUN_PROCESSORS}_${OMP_NUM_THREADS}.log
 mv TEST $test_dir/TEST.$i

done

Datura

Copy datura.sh as datura-testsuite.sh in simfactory/scriptfiles. Replace the mpirun command with the following:

cd @SOURCEDIR@

test_dir=TEST_$(date +%Y-%m-%d-%H%M%S) mkdir $test_dir

CONFIGNAME=$(ls configs|tail) # SimFactory should provide the configuration name

export CACTUS_STARTTIME=$(date +%s)

for i in 1 2 3; do

 case $i in
   1 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=1
     export OMP_NUM_THREADS=1;;
   2 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=2
     export OMP_NUM_THREADS=1;;
   3 )
     export CCTK_TESTSUITE_RUN_PROCESSORS=2
     export OMP_NUM_THREADS=2;;
 esac
 export CCTK_TESTSUITE_RUN_COMMAND="${MPIDIR}/bin/mpirun -v --mca btl openib,self --mca mpi_leave_pinned 0 -np \$nprocs -npernode $CCTK_TESTSUITE_RUN_PROCESSORS \$exe -L 3 \$parfile"
 make $CONFIGNAME-testsuite PROMPT=no
 cp TEST/$CONFIGNAME/summary.log datura__${CCTK_TESTSUITE_RUN_PROCESSORS}_${OMP_NUM_THREADS}.log
 mv TEST $test_dir/TEST.$i

done