CarpetX

The folllowing is the work done by S. Cupp on the CarpetX framework.

CarpetX Interpolator

Added the interface to connect Cactus' existing interpolation system to the CarpetX interpolator
AHFinder interpolation test was extended to compare the results from directly calling CarpetX's interpolator with the results from calling the new interface.

Added support for distrib=const arrays in CarpetX
Combined the structures for arrays and scalars into a single struct
All scalar code was extended to be able to handle both scalars and arrays
Added test in TestArray thorn to verify that array data is allocated correctly and behaves as expected when accessed

Overloaded the DynamicData function for use with CarpetX
Added the necessary data storage in the array group data to provide the dynamic data when requested
Added test in TestArray thorn to verify that dynamic data for grid functions, scalars, and arrays returns the correct data
Test also serves as basic test for read/write declarations, as all three variable types are written and read in this test

Multipole incorporated into cactusamrex repository
Test produces data that matches old code (be sure to use the same interpolator)
TwoPunctures thorn was temporarily changed to explicitly loop over indices instead of using loop macros
qc0 par file now runs, and all compile-time and runtime errors appear to have been resolved.
The simulation is quite slow, as S. Brandt's simulation was averaging ~300 seconds/iteration. After merging in the cuda branch, S. Cupp observed a simulation speed of ~200 seconds/iteration, which corresponds to ~2.8 M/day. This is unreasonably slow. The Toolkit paper needs a simulation time of ~300M to get the full merger. We need to reach a simulation speed that makes this viable before we can compare to previous simulations. To determine the source of the issue, we need to use gprof, cctk_timer_output, or other profiling tools to track down the slow part of the code.
3 runs were done: a run with Weyl/Multipole, a run without W/M, and a run without W/M that has tiling. These were simulated for 0.5M on Spine with 8 MPI processes. A summary of the timing data is provided below.

Simulation	Simulation Time	CCTK_INITIAL Time	CCTK_EVOL Time	CCTK_ANALYSIS Time
Full Run	25892.98	1929.70	12409.22	7577.40
No Weyl without Tiling	18077.91	1936.15	12573.32	578.31
No Weyl with Tiling	20105.39	1953.53	13959.25	643.85

It was expected that removing tiling would improve the runtime for the initial data; however, TwoPunctures already uses the CCTK_LOOP3_ALL structure instead of an explicit loop, so this didn't have a significant effect.
run simulation on Melete w/ 40 core

storage being always-on in CarpetX results in attempted regridding,etc. that causes validity failures. This is caused by the expected number of time levels not matching the actual number of time levels.
If there are too few cells, an unclear error appears which boils down to "assertion 'bc=bcrec' fails". This is because the cells are so large w.r.t. the number of points that multiple physical boundaries are within one cell. Identifying this as the problem and generating a different error message is recommended. Alternatively, the code could be altered to allow for these kinds of runs to function, though I am not sure that this would be worthwhile if it takes much effort/time.
CarpetX seems to use substantial resources for the qc0 test, and we are unsure of why. The amount of memory is several times higher than Roland's estimates for the usage with the given grid size.
The validity checking is incorrect for periodic boundary conditions. These boundary conditions are handled by AMReX, so no boundary conditions are run on the level of Cactus. Because it is all internal to AMReX, CarpetX never sets the boundaries to be valid. To fix this, any time the ghost zones are set to valid, the boundaries should also be set to valid. This should only be done when periodic boundary conditions are being used. Once this bug is fixed, the Weyl schedule.ccl should be reviewed as it contains hacks to bypass this bug.

SymmetryInterpolate isn't hooked up yet, but commented out code provides a starting point
Currently, CarpetX and Cactus both have parameters for interpolation order. As an example, qc0.par has to set "CarpetX::interpolation_order = 3" and "Multipole::interpolator_pars = "order=3" ". These should be condensed into a single parameter. Since individual thorns are setting their own interpolation order, I am assuming that different orders can be chosen for different thorns. If different variables have different orders of interpolation, the current implementation would break. Instead of using its own parameter, CarpetX should work with the existing infrastructure.
CarpetX's interpolate function doesn't return error codes, but historically there have been error codes for the interpolator. The new interpolate function should incorporate the old error codes to fully reproduce the functionality of the old interpolator
Interpolation interface should print out error codes for TableGetIntArray()
Distributed arrays are still not supported. It is unclear where these are used (at least to me). However, if they are needed, CarpetX will need to be extended to support them.
The DynamicData test revealed a bug with the scalar validity code. As such, we should consider whether we need more tests which specifically test the functionality of the valid/invalid code for the various types of variables. For example, tests for validating the poison routine, NaN checker, etc.

R. Haas resolved the issue with how CarpetX handled the difference between cell- and vertex-centered grid functions from old thorns. Incorrect default settings caused issues with interpolation, looping, etc. For example, TwoPunctures and Multipole required a hack where looping needed cctk_lsh[#]+1 instead of cctk_lsh[#]. Now that CarpetX/Cactus properly handles this data, old thorns should not need hacks to properly loop over grid functions.