Improved scheduling

Some months ago, we discussed possible ways to make scheduling in Cactus more straightforward for thorn writers. Please see Adding requirements to the Cactus scheduler for the work that was done before. We then had a telecon on 24-Oct-2011 (minutes?) in which we did some brainstorming and introduced the ideas to some new people.

Interested parties: Ian, Erik, Barry, Steve, Alexander, Peter, Oleg, Bruno M, Christian R, Frank

Current status

• There is a branch of the Cactus flesh which supports REQUIRES and PROVIDES directives in schedule.ccl files.

• There is a cut-down WaveToy thorn with very simple schedule requirements.

Possible plan

(including long-term vision. Please discuss!)

• Change REQUIRES and PROVIDES to READS and WRITES - we seem to have consensus that this makes more sense

• Define a semantics for accessing only parts of the domain, e.g. boundaries, symmetries. What domain parts should be supported? Should sync, prolongation, buffer zone prolongation, symmetries be applied automatically? Do we then just have interior, boundary, and everything? If so, note that interior | boundary != everything since we don't have an explicit notion for ghosts and buffer zones. Should there, in addition to a CCL syntax, be an API for declaring things at run-time? If so, should this API be more extensive than the CCL syntax? Probably best to write down, on a white board, the schedule for a parallel unigrid WaveToy with MoL.

• Set up a sequence of functions in CCTK_INITIAL with dependency information between variables. Make Carpet decide the scheduling order within CCTK_INITIAL based on these.

• Get evolution working

• Get boundary conditions working

• Make everything work for unigrid

• Make Carpet handle regridding/restricting/syncing/prolongating of all variables by calling scheduled functions which say they can write these variables.

• Add READS and WRITES information to all scheduled functions in the Einstein Toolkit - obviously this will be a large task and should be done bit-by-bit

• Check that the schedule that is derived is the same as before (except where it was wrong or ambiguous before - Frank mentioned that he once tried to change Cactus to schedule functions in the reverse order where it is currently ambiguous, and this led to errors, so there are probably some thorns which do not correctly specify orderings).

• Deprecate unneeded schedule groups and "before" and "after" and transition entirely to the new dependency-based scheduling (one year from now?)

Issues/thoughts

• Think about how to deal with MoL. Should we first try to get evolution working without MoL?

• We almost certainly will need to have some overlap between the old scheduler and the new scheduler. For this it probably makes sense to allow (and ignore) the new keywords in the flesh even in trunk so that we don't need a separate branch for every thorn that we add the keywords to. In Carpet, we could have some parameters to control the scheduler - i.e. to use the old one or the new one.

• What shall we do with PUGH?

• Do thorns need to declare dependencies on all Cactus variables that they access? e.g. grid scalars, grid arrays, spherical surfaces etc? I think yes. What about flesh variables: cctk_nghostzones etc? I think no.

• Is it too much effort for a thorn writer to declare all the dependencies in the schedule.ccl file?

• We cannot derive the dependencies automatically from the source file. For strict checking, we could make variables which we have not asked for either null pointers or not available at the compiler level - I prefer the latter but the former would be easier to implement first.

• Eventually we can check writes as well as reads by doing checksumming of data after a function is called. This is mostly important for checking that different regions of the grid have been written according to the scheduling.

• Idea: can we eventually eliminate all bins and groups and "before/after" directives, and drive the scheduling entirely based on data dependency? Would need to have "contexts" for variables. e.g. saying that a function writes gxx_initial, and MoL then reads gxx_initial and writes gxx.

Work plan

What would we like to achieve during this workshop? Just discussion, or also some coding?