Test Failures

From Einstein Toolkit Documentation
Jump to: navigation, search

As of 13-Mar-2017, several tests fail on the Jenkins build system, but pass on other systems. All tests here were run with code checked out on or after 13-Mar-2017.

Some details are in ticket 2019.

System /proc/cpuinfo /etc/issue gcc --version
Jenkins build node Intel Core Processor (Haswell) Ubuntu 16.04 LTS gcc (Ubuntu 5.3.1-14ubuntu2) 5.3.1 20160413
login.barrywardell.net Intel Core Processor (Haswell) Ubuntu 16.04.2 LTS gcc (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Archimedes Intel(R) Xeon(R) CPU E5-2630 v4 Ubuntu 14.04.5 LTS gcc (Ubuntu 4.8.4-2ubuntu1~14.04.3) 4.8.4
Test Jenkins
build node
login (ubuntu.cfg) login (build.cfg) login (ubuntu.cfg
without march=native
login (ubuntu.cfg
without march=native,
without fast-math
Archimedes (ubuntu.cfg)
CT_MultiLevel.boostedpuncture/2procs Fail Fail Fail Fail Fail
SphericalHarmonicReconGen.SpEC-dat-test/1procs Fail
SphericalHarmonicReconGen.SpEC-h5-test/1procs Fail
SphericalHarmonicRecon.regression_test/2procs Fail
SphericalHarmonicReconGen.SpEC-dat-test/2procs Fail
SphericalHarmonicReconGen.SpEC-h5-test/2procs Fail
GRHydro.GRHydro_test_shock_weno/1procs Fail Fail Pass Fail Pass Pass
GRHydro.GRHydro_test_shock_weno/2procs Fail


SphericalHarmonicReconGen

  • one failure was due to a bug in NaNChecker introduced in 9fc4866
  • the other uses HDF5 incorrectly which however does not seem to be the root cause of the failure

WENO Test

  • Roland tried multiple different resolutions (up to 8 times more and 16 times the time resolution, as well as 2 times bigger and offset from 0 grids) none of which prevented NaNs to from on my laptop
  • next thing to try is to find out which is the exact optimization options that causes the NaN
  • the option responsible for test failure is -fassociative-math
  • the affected file is GRHydro_WENOReconstruct.cc and in particular the rescaling of beta[123] when adaptive_epsilon is used with very small (which is the default) weno_eps