Improved scheduling

From Einstein Toolkit Documentation
Revision as of 10:55, 1 November 2011 by Sbrandt (talk | contribs)
Jump to: navigation, search

Some months ago, we discussed possible ways to make scheduling in Cactus more straightforward for thorn writers. Please see Adding requirements to the Cactus scheduler for the work that was done before. We then had a telecon on 24-Oct-2011 (minutes?) in which we did some brainstorming and introduced the ideas to some new people.

Interested parties: Ian, Erik, Barry, Steve, Alexander, Peter, Oleg, Bruno M, Christian R, Frank

Current status

  • There is a branch of the Cactus flesh which supports REQUIRES and PROVIDES directives in schedule.ccl files.
  • There is a cut-down WaveToy thorn with very simple schedule requirements.

Possible plan

(including long-term vision. Please discuss!)

  • Change REQUIRES and PROVIDES to READS and WRITES - we seem to have consensus that this makes more sense
  • Define a semantics for accessing only parts of the domain, e.g. boundaries, symmetries. What domain parts should be supported? Should sync, prolongation, buffer zone prolongation, symmetries be applied automatically? Do we then just have interior, boundary, and everything? If so, note that interior | boundary != everything since we don't have an explicit notion for ghosts and buffer zones. Should there, in addition to a CCL syntax, be an API for declaring things at run-time? If so, should this API be more extensive than the CCL syntax? Probably best to write down, on a white board, the schedule for a parallel unigrid WaveToy with MoL.
  • Set up a sequence of functions in CCTK_INITIAL with dependency information between variables. Make Carpet decide the scheduling order within CCTK_INITIAL based on these. This could be possible in Cactus, by just adding the currently supported BEFORE and AFTER options to the schedule information based on the values of the READS and WRITES options. Carpet/driver would then insert its own calls to so sync/prolongate etc, but the ordering wouldn't change. Could provide a version of ScheduleTraverse which took a list of variables (or parts of variables) that need to be filled. This would run a passed callback function as it currently does, but only on those functions which are needed to compute those variables.
  • Get evolution working
  • Get boundary conditions working
  • Make everything work for unigrid
  • Make Carpet handle regridding/restricting/syncing/prolongating of all variables by calling scheduled functions which say they can write these variables.
  • Add READS and WRITES information to all scheduled functions in the Einstein Toolkit - obviously this will be a large task and should be done bit-by-bit
  • Check that the schedule that is derived is the same as before (except where it was wrong or ambiguous before - Frank mentioned that he once tried to change Cactus to schedule functions in the reverse order where it is currently ambiguous, and this led to errors, so there are probably some thorns which do not correctly specify orderings).
  • Deprecate unneeded schedule groups and "before" and "after" and transition entirely to the new dependency-based scheduling (one year from now?)

Issues/thoughts

  • Think about how to deal with MoL. Should we first try to get evolution working without MoL?
  • We almost certainly will need to have some overlap between the old scheduler and the new scheduler. For this it probably makes sense to allow (and ignore) the new keywords in the flesh even in trunk so that we don't need a separate branch for every thorn that we add the keywords to. In Carpet, we could have some parameters to control the scheduler - i.e. to use the old one or the new one.
  • What shall we do with PUGH?
  • Do thorns need to declare dependencies on all Cactus variables that they access? e.g. grid scalars, grid arrays, spherical surfaces etc? I think yes. What about flesh variables: cctk_nghostzones etc? I think no.
  • Is it too much effort for a thorn writer to declare all the dependencies in the schedule.ccl file?
  • We cannot derive the dependencies automatically from the source file. For strict checking, we could make variables which we have not asked for either null pointers or not available at the compiler level - I prefer the latter but the former would be easier to implement first.
  • Eventually we can check writes as well as reads by doing checksumming of data after a function is called. This is mostly important for checking that different regions of the grid have been written according to the scheduling.
  • Idea: can we eventually eliminate all bins and groups and "before/after" directives, and drive the scheduling entirely based on data dependency? Would need to have "contexts" for variables. e.g. saying that a function writes gxx_initial, and MoL then reads gxx_initial and writes gxx.
  • Identifying different parts of the grid: inner and outer ghost points - cf CaKernel
  • Do we want to have dependencies on functions as well as data? Does this make some things simpler?
  • How can we "insert" functions into "pipelines". e.g. adding noise or adding some other perturbation, which might not be aware of each other. How to resolve the order of these? The user is the only one who knows what order they should be run in. This has to be provided in the parameter file.
  • Should some functions just be marked as "filters"?
  • Can we say that we only need e.g. the outermost point of the grid to be synced because our particular algorithm can fill in some of the ghost points? (optimisation)
  • Automatically determine number of ghost/boundary points
  • Standard method for application thorns to do looping over different parts of the grid
  • Can the schedule change at runtime? When would be a good time to schedule it? Same time that recovery can happen - important to consider interaction with Berger-Oliger timestepping.
  • Checksumming won't work for determining if somebody has written data back the same as was there. Maybe invalidate data first (poison). What if you read and write the same variable? Then we just don't have any way to check correctness and have to assume the code is correct.
  • Can we have dependencies on parameters? e.g. if a parameter is steered, some other functions might need to be rerun to update things (e.g. regridding controlled by parameter). Don't want to list all the parameters which are used as input for a function.
  • How do we say which spherical surface we write, when this is determined at run-time from a parameter? Can we put parameters into these WRITES statements? e.g. WRITES: sphericalfurface::sf_centroid_x[ah_sf] where ah_sf is a parameter?
  • Completely separate idea: Named spherical surfaces
  • Christian introduced SphericalSlice

Work plan

What would we like to achieve during this workshop? Just discussion, or also some coding? 

    SCHEDULING
    Problems
        * Complicated to get it to work with mesh refinement
        * Verbose, you have to type a lot of things
    Aim:
        * User should just specify what function sets/does
        * Add information to the schedule CCL file to say
          what variables are read/written
        * Fewer bins
        * Look at what DAGMan does
        * Look at Uintah http://www.sci.utah.edu/research/computing/271-uintah.html
        * Create variables or modify variables
        * If you have a hybrid,
            * variable read/write provides a check
            * can determine sync/prolong
            * automatic memory allocation/freeing
    BUILD SYSTEM
    * Top two priorities:
        1) Python front end
        2) Understand what the current system does
    * Grammars for CCL
        * enables other programs (e.g. Kranc) to access contents
        * makes things less ambiguous, more professional
    * Document CACTUS_CONFIGS_DIR
        * Should there be an ARRANGEMENTS_DIR or ARRANGEMENTS_PATH?
    * Can the build system be simpler?
        * Python front-end: Ticket 332 on Cactus Makefile replacement
            * Do this first becaues it would be easy
        * Generate master make
    * Alternatives to Make?
        * ant - Alexander will study
        * cmake
        * maven - Has a C/C++ plugin
    * Progress indicator?
    * Cactus test suite interface
        * list test suites
        * run just one test
        * Need non-interactive test suite mode
        * Ian believes this shouldn't just be in simfactory
    * Maybe a lot of this could be fixed with better documentation
        * Probably no one understands all of it
        * Heavily tied into autotools
Retrieved from "https://docs.einsteintoolkit.org/et-docs/index.php?title=Improved_scheduling&oldid=2793"