October 22nd

Scheduler

Steve has worked on a Piraha java/c++ parser for the new data-driven scheduler specifications. He has run some tests, and run some cactus testsuites. Things seem to work. There's a strange COMM clause in some schedule.ccl files, that is old and should probably be removed.

At the moment, this only works on the schedule.ccl files, but it's in principle simple to extend it to other .ccl files. interface.ccl is the next obvious choice as it's the most complicated one and the function syntax should be improved anyway. Piraha already improves on the schedule.ccl grammar by allowing multiline support, better error messages. Steve can give a demo of this sometime this week.

Roland and Erik have also worked on the data-driven scheduler. This has mostly involved adding the READ/WRITE clauses to the schedule.ccl files, and then add consistency checks in Carpet to make sure that the clauses enforce the correct behavior, currently implemented via the old scheduling scheme. Slowly one would start switching thorns to the new method.

The new method entails grid-part specifications for the grid functions, so that it can also tell Carpet how to form its loops and where to apply functions.

A related problem is whether to keep the BEFORE/AFTER clauses we currently have. In principle these are bad since they refer to function names and the schedule's behavior when certain functions are not active is undefined. Removing them is rather complicated though, as sometimes actions have to be executed on the same data, so that the argument of READ and WRITE is the same and one has to specify additional structures to disambiguate. Perhaps one could simply have blocking BEFORE/AFTER to enforce that the function that is referred to is actually executed.

Motivations: to simplify syncs and storage, but also for GPU programming and help with scalability, as fine-grained data flow would enable cores to work asynchronously. This can also help eliminate things like DECLARE_CCTK_ARGUMENTS, which is sometimes rather wasteful, and tune the passed arguments based on the data requirements that have been specified.

In order to implement functions that act on specific parts of the grid, one needs macros to access data selectively. Erik: these already exist in the flesh (cctk_loop.h). LoopControl uses and enhances these. Should the grid specifications be improved? Currently there are a few options, that are specified after the variable names, in round parentheses. More than one region can be specified in a semicolon-separated list.

This discussion is better resumed tomorrow, once Roland arrives.

Make system

The autoconf system is used in a slightly unusual way in Cactus, with several layers of generated Makefiles.

A more serious issue is that with the missing Fortran compiler. How can we fix this? Cactus should search for things that it doesn't find. Should there be an ExternalLibraries thorn for this? How is the Fortran compiler different, say, from MPI?

A solution would be to provide Cactus as a package, that can just be handled by package managers on most Linux systems, including supercomputers. What exactly should be in the package? Local optionlists? The entire ET source tree? Some other communities provide globally-installed versions of their codes for users to use out of the box. Thorns in ExternalLibraries could be taught to recognize the presence of such a package and inherit options from it.

Steve's work (ubiqis) with config and thorn directories in separate locations from the source tree introduces some of the required infrastructure to run Cactus in this way. However, these changes have never been committed. Steve can look for this material and get it into the repository.

Optionlists should still have to be maintained though, for those systems that do not have package managers. On this topic, we should have compiler-specific, rather than architecture-specific, optionlists, as certain options only really apply to certain compilers. Should these go in ExternalLibraries thorns? Should the compiler type be detected at config time? Through autoconf?