Testsuite Machines

This page contains notes and instructions for people running the ET testsuites on various different machines. If you have experience running testsuites on a machine which is not listed here, please consider adding some information which might help others (or yourself!) in future.

General

To check out the ET:

 mkdir etrelease
 cd etrelease
 curl -O https://github.com/gridaphobe/CRL/raw/master/GetComponents 
 chmod a+x GetComponents
 ./GetComponents --root=. -a https://svn.einsteintoolkit.org/manifest/trunk/einsteintoolkit.th
 cp udb.example.pm udb.pm

Replace "redshift" with the name of your local machine, unless you are running on one of the supported machines.

Replace YOUR_LOGIN, YOUR_EMAIL_ADDRESS and YOUR_ALLOCATION as appropriate.

Set sourcebasedir for your local machine to be the directory containing etrelease (do not set it to etrelease itself).

Set your username on each machine:

   set_option 'MACHINE'  , 'user', 'USERNAME';

See the machine-specific notes below for any additional steps for each machine.

 touch par/testsuite.par
 sim sync <machine>
 sim build --thornlist manifest/einsteintoolkit.th --scriptfile <machine>-testsuite.sh

 sim remote <machine> create-submit par/testsuite.par NSLOTS 6:00:00

This is a dummy parameter file.

When the job is finished, you should have the required <machine>_i_j.log files in machine:etrelease/*.log. The testsuite output will also be stored in machine:TEST_XXX, where XXX is constructed from the date at which the job was run.

Update the testsuite status page by adding the log files to the release-info repository:

 svn co https://svn.einsteintoolkit.org/www/release-info
 cd release-info
 scp machine:etrelease/*.log .

Then commit the new/updated files.

Machines

Kraken

Edit simfactory/mdb.pm

 -        myproxy-logon -p 7514 -s myproxy.teragrid.org -T -l @USER@ -o @SOURCEDIR@/.globus/proxy-teragrid
 +        myproxy-logon -p 7514 -s myproxy.teragrid.org -T -l <username> -o @SOURCEDIR@/.globus/proxy-teragrid

where <username> is your user name on Kraken. This is related to Ticket #381

Change the sourcebasedir in simfactory/udb.pm to be under /lustre/scratch. This is necessary because the Cactus directory must be visible from the compute node when running the tests (to see the parameter files, testsuite reference output and test.ccl files).

   set_option 'kraken', 'sourcebasedir', '/lustre/scratch/USERNAME';

Copy kraken.sh as kraken-testsuite.sh in simfactory/scriptfiles. Replace the aprun command with the following:

cd @SOURCEDIR@

export CCTK_TESTSUITE_RUN_COMMAND="aprun -n \$nprocs -d 1 \$exe \$parfile"

test_dir=TEST_$(date +%Y-%m-%d-%H%M%S)
mkdir $test_dir

CONFIGNAME=$(ls configs|tail) # SimFactory should provide the configuration name

for i in 1 2 3; do
  case $i in
    1 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=1
      export OMP_NUM_THREADS=1;;
    2 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=2
      export OMP_NUM_THREADS=1;;
    3 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=2
      export OMP_NUM_THREADS=2;;
  esac
  make $CONFIGNAME-testsuite PROMPT=no
  cp TEST/$CONFIGNAME/summary.log kraken__${CCTK_TESTSUITE_RUN_PROCESSORS}_${OMP_NUM_THREADS}.log
  mv TEST $test_dir/TEST.$i
done

Datura

Copy datura.sh as datura-testsuite.sh in simfactory/scriptfiles. Replace the mpirun command with the following:

cd @SOURCEDIR@

test_dir=TEST_$(date +%Y-%m-%d-%H%M%S)
mkdir $test_dir

CONFIGNAME=$(ls configs|tail) # SimFactory should provide the configuration name

export CACTUS_STARTTIME=$(date +%s)

for i in 1 2 3; do
  case $i in
    1 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=1
      export OMP_NUM_THREADS=1;;
    2 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=2
      export OMP_NUM_THREADS=1;;
    3 )
      export CCTK_TESTSUITE_RUN_PROCESSORS=2
      export OMP_NUM_THREADS=2;;
  esac
  export CCTK_TESTSUITE_RUN_COMMAND="${MPIDIR}/bin/mpirun -v --mca btl openib,self --mca mpi_leave_pinned 0 -np \$nprocs -npernode $CCTK_TESTSUITE_RUN_PROCESSORS \$exe -L 3 \$parfile"
  make $CONFIGNAME-testsuite PROMPT=no
  cp TEST/$CONFIGNAME/summary.log datura__${CCTK_TESTSUITE_RUN_PROCESSORS}_${OMP_NUM_THREADS}.log
  mv TEST $test_dir/TEST.$i
done