ET Workshop Fall 2011 Summary Cactus

Improved scheduling

(For full notes see Improved scheduling).

Aim: Simplify correct scheduling in Cactus

Conclusions

• Currently very difficult and complicated to ensure that boundary conditions (symmetry, physical, interprocessor, refinement) are applied to the correct parts of the correct variables at the correct time. Also hard to ensure correct ordering between scheduled functions.

• Propose that information is added to schedule.ccl to say what parts of what variables each function reads and writes. This allows some of the scheduling to be automated and some of the current schedule groups (PostPostInitial etc) to be eliminated. Any remaining ambiguities will be handled by BEFORE and AFTER clauses, as currently.

• Have decided on a set of disjoint and covering "parts" of the grid (can be changed if necessary): Interior, PhysicalBoundary, SymmetryBoundary, InterprocessorBoundary, RefinementBoundary. Functions can claim to set any combination of these.

• READS and WRITES clauses in schedule.ccl will refer to groups only, not variables.

• Thorns can declare READS and WRITES at runtime - this is necessary for MoL, Dissipation, Noise etc where this cannot be known at compile time.

• MoL might be handled as a special case.

• MoL can infer which parts of the grid should be integrated by detecting which parts of the RHS variables have been set using poisoning. MoL will infer overall READS and WRITES from those of the functions scheduled in it.

• We will work incrementally, adding first READS and WRITES support to the flesh and annotations to thorns. Carpet/Cactus can then perform checking. We can start to replace certain scheduling with automatically derived information. Aim is to do this as much as possible.

• Carpet will check that the correct parts of variables have been written using checksumming. It is not possible to check READS for parts of grid functions, but we could set unused grid variable pointers to NULL, or just not pass them as arguments.

• We will still need Cactus bins such as INITIAL, as this is the easiest way to ensure that initial data is computed only initially.

• Have decided **not** to use aliases or tagging of variables to deal with difficult scheduling issues. It adds complication and could just as easily be done using the existing BEFORE and AFTER options. This could be revisited later if we wanted to have a "pure" data-driven schedule.

• Some scheduling cannot be determined at compile time because the thorn-writers don't know about each others' thorns. Could have a flesh parameter such as dependencies = Noise < Perturb, <fn2> < <fn3>, ... so that the end-user can control how ambiguous schedules should be resolved (add noise before the perturbation).

Concrete plans

• Will provide a flesh API for getting the schedule information needed (maybe this already exists)

• Will write a new thorn (which may go into the flesh eventually) which returns the functions needed to be run in order for updating particular variables. This can be called from Carpet when it needs to update particular variables.

Still to be decided

• Would like to eliminate the need to declare Carpet modes (global, local, level etc) in schedule.ccl files, but have not discussed how to do this.

• Would like to avoid the need to declare number of timelevels used - this should be derivable from the READS and WRITES information.

• State dependencies on parameters to automate/check steering issues?