ET Workshop Fall 2011 Summary Cactus

General Elliptic Solver

Discussion and conclusions

• There is a distinction between inversion methods and relaxation methods; the former use matrix inversion to solve a linear system (possibly coming from the linearization of a non-linear problem), whereas the latter reduce the residuals to zero by implementing the corresponding parabolic equation and let the solution relax to a static one. PETSc has some infrastructure for relaxation methods (namely, multigrid solvers), but has much more in the way of inversion methods (along with linearization methods). Additionally, AMR is easy to cope with in the inversion methods, where the grid (in real space or in frequency space) is mapped to a matrix and its "connectivity" is irrelevant, but is harder to have around in relaxation methods, where the jacobians of all the involved operators (including prolongation, ...) have to appear; [ES: this seems wrong -- PETSc requires the grid connectivity (Jacobian), which is the complex part]
• Erik has some experience implementing solvers based on the PETSc library, and has confidence that this method would yield robust tools as long as one is willing to code up all the equation jacobians; these will be variably complicated depending on the type of mesh refinement, i.e. relatively simple for unigrid or old-school multipatch, more complicated for new multipatch and AMR. This means one could complete this project step-by-step, progressively adding AMR capabilities by coding up the appropriate jacobians;
• TexMEx, a multigrid solver with AMR capabilities, is currently available in Cactus. The integration with Cactus (and Carpet) is currently minimal: TexMEx is used as a black box that produces the result (with its own data structures, etc.) and only at the end communicates it to Cactus. This prevents the framework from seeing inside its algorithm, which is not ideal for debugging, and additionally makes it hard to extend the package, e.g. by letting the user specify a callback function to calculate the residual. As of now, TexMEx implements the hamiltonian constraint, which doesn't help who (Steve, Ian) is interested in solving other types of equations.
• Peter's NoExcision thorn uses [Conjugate Gradient] methods with finite differencing. This seems like a good method to use and should be easy to implement. A generic infrastructure could be written for this, where a user provides just the RHS function as in MoL. This could also be done in Kranc. Not sure what the drawback would be - maybe it would not be efficient enough, or maybe there are subtleties we are not aware of due to lack of knowledge. Assuming it would work with mesh-refinement, but have no knowledge of any possible problems. [ES: This does not use a preconditioner, so the convergence may be slow unless the residual is formulated carefully.]

Plan

• Take LSUPETSc as a basis, and try to add the missing pieces to make it work with the grids we're interested in;
• Check with the TexMEx developers what their roadmap is for integration with Carpet, and see if we can make this tool solve generic equations.
• Implement a conjugate-gradient solver in Kranc and then try to generalise to a generic elliptic interface.

Build system

• Top two priorities:
  • Python front end
  • Understand what the current system does
• Grammars for CCL
  • enables other programs (e.g. Kranc) to access contents
  • makes things less ambiguous, more professional
• Document CACTUS_CONFIGS_DIR
• Should there be an ARRANGEMENTS_DIR or ARRANGEMENTS_PATH?
• Can the build system be simpler?
  • Python front-end: Ticket 332 on Cactus Makefile replacement
    • Do this first becaues it would be easy
    • Generate master make
  • Alternatives to Make?
    • ant - Alexander will study
    • cmake
    • maven - Has a C/C++ plugin
• Progress indicator?
• Cactus test suite interface
  • list test suites
  • run just one test
  • Need non-interactive test suite mode
  • Ian believes this shouldn't just be in simfactory
  • Ian wants XML output of test results for automated test system - Ian has started working on this but not much work done yet
• Maybe a lot of this could be fixed with better documentation
  • Probably no one understands all of it
  • Heavily tied into autotools

GPUs

• CaKernel being developed for CUDA and OpenCL;
• Kranc could generate the equations and boilerplate parts that are needed;
• Would need to improve the way that finite differences are expressed in these equations - precompute;
• Use existing CaKernel templates for optimised operations.

Improved scheduling

(For full notes see Improved scheduling).

Aim: Simplify correct scheduling in Cactus

Conclusions

• Currently very difficult and complicated to ensure that boundary conditions (symmetry, physical, interprocessor, refinement) are applied to the correct parts of the correct variables at the correct time. Also hard to ensure correct ordering between scheduled functions.

• Propose that information is added to schedule.ccl to say what parts of what variables each function reads and writes. This allows some of the scheduling to be automated and some of the current schedule groups (PostPostInitial etc) to be eliminated. Any remaining ambiguities will be handled by BEFORE and AFTER clauses, as currently.

• Have decided on a set of disjoint and covering "parts" of the grid (can be changed if necessary): Interior, PhysicalBoundary, SymmetryBoundary, InterprocessorBoundary, RefinementBoundary. Functions can claim to set any combination of these.

• READS and WRITES clauses in schedule.ccl will refer to groups only, not variables.

• Thorns can declare READS and WRITES at runtime - this is necessary for MoL, Dissipation, Noise etc where this cannot be known at compile time.

• MoL might be handled as a special case.

• MoL can infer which parts of the grid should be integrated by detecting which parts of the RHS variables have been set using poisoning. MoL will infer overall READS and WRITES from those of the functions scheduled in it.

• We will work incrementally, adding first READS and WRITES support to the flesh and annotations to thorns. Carpet/Cactus can then perform checking. We can start to replace certain scheduling with automatically derived information. Aim is to do this as much as possible.

• Carpet will check that the correct parts of variables have been written using checksumming. It is not possible to check READS for parts of grid functions, but we could set unused grid variable pointers to NULL, or just not pass them as arguments.

• We will still need Cactus bins such as INITIAL, as this is the easiest way to ensure that initial data is computed only initially.

• Have decided **not** to use aliases or tagging of variables to deal with difficult scheduling issues. It adds complication and could just as easily be done using the existing BEFORE and AFTER options. This could be revisited later if we wanted to have a "pure" data-driven schedule.

• Some scheduling cannot be determined at compile time because the thorn-writers don't know about each others' thorns. Could have a flesh parameter such as dependencies = Noise < Perturb, <fn2> < <fn3>, ... so that the end-user can control how ambiguous schedules should be resolved (add noise before the perturbation).

• [ES: Need to think about global/level/local mode etc., e.g. interpolation needs data from all levels, RHS just from the current level]

Concrete plans

• Will provide a flesh API for getting the schedule information needed (maybe this already exists)

• Will write a new thorn (which may go into the flesh eventually) which returns the functions needed to be run in order for updating particular variables. This can be called from Carpet when it needs to update particular variables.

Still to be decided

• Would like to eliminate the need to declare Carpet modes (global, local, level etc) in schedule.ccl files, but have not discussed how to do this.

• Would like to avoid the need to declare number of timelevels used - this should be derivable from the READS and WRITES information.

• State dependencies on parameters to automate/check steering issues?