From Einstein Toolkit Documentation
Jump to: navigation, search

The purpose of this page is to explain how to properly implement the features provided by the PreSync update to a thorn, as well as how to do this while maintaining backward compatibility.


The PreSync update (publication) introduces the concept of data-dependent scheduling to Cactus and Carpet. While not fully data-dependent, this update specifically automates the scheduling of ghost zone synchronization and the application of boundary conditions. The upcoming driver thorn CarpetX is designed from the ground up with PreSync in mind, and thorns will be required to be PreSync-compatible to use CarpetX. Even in Carpet, automating the scheduling of inter-processor communication makes the writing of thorns easier; thorns using PreSync do not need to use the SYNC statements or manually schedule ApplyBCs so long as they are running with PreSync active. Of course, for backward compatibility, thorns will still need to provide scheduling information for these to still properly function with PreSync disabled.

In the following sections the way to implement PreSync is outlined, starting with new thorns without backward compatibility. Following this is a section detailing how one implements these new features in an older thorn while still maintaining backward compatibility (or writing a new thorn which can work with both).

New Thorns

Unsurprisingly, writing a new thorn which will only work with PreSync active is simpler than providing backward compatibility. For most thorns, this is a relatively simple matter. There are several places at which new thorns will diverge from old thorns.


Cactus user guide (authorative) documentation.

Read/Write Declarations

The schedule.ccl file will have several changes. First, each scheduled function must declare what variables it reads and writes. These declarations take the general form of

schedule func at bin
  READS: thorn::variable/group(region)
  WRITES: thorn::variable/group(region)

Every variable (grid functions, scalars, arrays, etc.) which the function func accesses must be included in this list. The variable/group usually gives the variable that is accessed. If the entire group is accessed, then the group name can be given instead. The region can be either interior, boundary, or everywhere. If a variable is read everywhere, this means that the grid function must have its ghost zones synchronized and boundary conditions applied. If this has not yet happened by the time func runs, then PreSync will automatically schedule these operations. Similarly, writing to the interior means that the ghost zones are no longer valid and must be synchronized the next time they are needed. Naturally, non-distributed variables (i.e. anything other than grid functions) have no interior and should only be read or written everywhere. These declarations can be simplified in several ways. First, the region is optional. If no region is given, then READs assume everywhere and WRITEs assume interior. Second, the "thorn::" part can be dropped for variables from the thorn that the schedule.ccl is from. For example, the ML_BSSN thorn could schedule something of the form

schedule func at bin
  READS: TmunuBase::stress_energy_tensor

where Xt1 is implicitly ML_BSSN::Xt1 and stress_energy_tensor provides READ access to every variable in the group stress_energy_tensor from the thorn TmunuBase.

Many functions have many variables accessed in the same way. In addition to using group names instead of variable names, several other features can simplify these expressions. First, a single line can give several variables/groups with the same access type. Again pretending we are scheduling func in ML_BSSN:

schedule func at bin
  READS: TmunuBase::stress_energy_scalar(everywhere), TmunuBase::stress_energy_vector(everywhere), TmunuBase::stress_energy_tensor(everywhere)
  WRITES: Xt1(interior), Xt2(interior), Xt3(interior), ML_mom(interior)

Note that we have mixed group and variable names in the WRITES declarations. This is perfectly fine. The one case in which group and variable names cannot be used in combination is if some variables in a group are read and others are written. Even if all of them are read/written, the group name cannot currently be used if the other declaration type only has some of the variables. They must be explicitly given as variable names in this case.

Another stylistic option for long lists of READS or WRITES is to extend them to multiple lines. Line continuation is automatic, as seen in the example below.

schedule func at bin
  READS: TmunuBase::stress_energy_scalar(everywhere),
  WRITES: Xt1(interior), Xt2(interior), Xt3(interior),

This can allow for slightly more readable scheduling in some cases, as variables can be grouped into a single READ/WRITE without enforcing that they are on a single line.

As an aside, one can, of course, have multiple implicit declarations on the same line:

schedule func at bin
  READS: TmunuBase::stress_energy_scalar, TmunuBase::stress_energy_vector, TmunuBase::stress_energy_tensor
  WRITES: Xt1, Xt2, Xt3, ML_mom

In theory, one could mix together explicit and implicit declarations on the same line; however, this could be ambiguous to someone looking at the scheduling. For example, it is not obvious in

schedule func at bin
  READS: var1(interior), var2
  WRITES: var3(everywhere), var4

what the regions are for var2 and var4. They do take on the implicit declaration of `everywhere' for reads and `interior' for writes, but this is stylistically undesirable. As such, it is recommended to not mix different regions of validity on the same line for clarity.

The second feature is for previous timelevels. In Cactus, the previous timelevels are accessed with an "_p" appended to the end of the variable name. Functions which access previous timelevels of variables must also declare these as separate variables. In the case that one needs access to the previous timelevels for an entire group of variables, the "_p" can instead be appended to the group name in the scheduling:

schedule func at bin
  READS: var1(everywhere), var1_p(everywhere), var1_p_p(everywhere)
  WRITES: group2(interior), group2_p(interior), group2_p_p(interior)

Important note: previously, there has been no strict enforcement of the public/protected/private status of variables in Cactus. However, the new declarations respect these designations. Therefore, any thorn which needs to access variables from other thorns needs to properly declare inherit or friend status of other thorns in the interface.ccl.

Selecting Boundary Conditions

In the old system, boundary conditions must be set via the Boundary_Select*ForBC() functions, after which the ApplyBCs group would be scheduled. These would be scheduled whenever BCs needed to be applied, and the BC selection would be cleared at the end of ApplyBCs. This is because the boundary selection not only told the Boundary thorn which BCs to apply to which variables but also determined which variables needed boundary conditions applied. With PreSync, this is much simpler. The boundary conditions are set via the Driver_Select*ForBC() functions, which take the same arguments as the old Boundary_Select*ForBC() functions. These functions are scheduled in the Driver_BoundarySelect group, which runs once at startup. PreSync saves these settings and uses them whenever BCs need to be applied. The ApplyBCs group never needs to be scheduled since BC application is automatic.

To summarize, boundary conditions for PreSync only requires a single function

schedule Thorn_SelectBCs in Driver_BoundarySelect
} "Selects boundary conditions for the variables in Thorn."

which sets all the boundary conditions for the variables from that thorn using Driver_Select*ForBC().

Source Code

New Macros

The old flesh provided the macro DECLARE_CCTK_ARGUMENTS for getting access to the variables declared in the interface.ccl of the thorn. PreSync provides new macros which are generated from the read/write declarations in the schedule.ccl. These take the form DECLARE_CCTK_ARGUMENTS_func where func is the name of the scheduled function. These new macros provide error-checking by ensuring that only the variables declared in the schedule.ccl can be accessed in the function. In addition, variables which are read but not written are declared as const, so any attempt to write them will also cause a compile-time error. Any functions called by the scheduled function must, of course, respect this constness, further ensuring that the read/write declarations aren't violated even by internal functions called by the parent scheduled function.

There are some subtleties in the new macros, however. For internal functions, it is sometimes far easier to simply use the macro DECLARE_CCTK_ARGUMENTS rather than explicitly passing all the variables in the argument list of the function. If the number of grid functions needed is not very large, then simply passing them will easily maintain the sanity check provided by the new macros. If the function just needs access to the various loop variables (e.g. cctk_lsh), then the macro _DECLARE_CCTK_ARGUMENTS provides access to these without any of the thorn's grid functions. If an internal function is only called by a single scheduled function, then the scheduled function's macro can also be used in the child function. However, doing so will lose some of the certainty about whether the read/write declarations are truly correct, and the thorn writers must take more of this burden upon themselves.

Runtime-dependent Reads/Writes

While this is (hopefully!) not a common occurrence, some thorns have to have read/write declarations decided at runtime. As an obvious example, IO thorns or MoL (Method of Lines thorn) must decide what they read and write at runtime. Some analysis thorns will also do this if they compute quantities of variables given at runtime. In wvuthorns_diagnostics, VolumeIntegrals_vacuum has parameters to tell the thorn what other thorn is providing the constraints for the simulation. These parameters will then change which variables VolumeIntegrals_vacuum is using in its calculations. To handle such a case, the function Driver_RequireValidData() provides the functionality to enforce READs dynamically. This function is provided by the driver (such as Carpet) and must be put in the interface.ccl. This function is declared as

   const cGH* cctkGH,
   CCTK_INT const * const vars,
   CCTK_INT const * const tls,
   CCTK_INT const nvars,
   CCTK_INT const * wheres)

where vars, tls, and wheres are all arrays of length nvars. vars contains the variable index of each variable that is dynamically read, tls contains the associated timelevel for each variable, and wheres contains the associated region (interior, everywhere, etc.) for each variable. Similarly, the function Driver_NotifyDataModified() provides functionality to dynamically assign WRITEs, and it takes the same arguments as Driver_RequireValidData().

Other Concerns

Respecting constness of read-only variables

There are some rare cases where more changes must be made. As an example, the thorn IllinoisGRMHD defines the struct

struct gf_and_gz_struct {
  CCTK_REAL *gf;
  int gz_lo[4],gz_hi[4];

In most cases, this is fine. If the grid function *gf is read-only, then one could simply define const gf_and_gz_struct. Unfortunately, part of the code required changing the values of gz_lo and gz_hi for a read-only *gf. To solve this without giving incorrect read/write information, it required defining a new struct

struct const_gf_and_gz_struct {
  const CCTK_REAL *gf;
  int gz_lo[4],gz_hi[4];

While rare, one should still be aware of the possible pitfalls of having variables declared as const in PreSync.

Proper use of the checkpoint tag

Cactus provides checkpointing and restart as one of its basic features, and a part of this is the option to turn off checkpointing for groups. However, many thorns do not turn off checkpointing for variables which do not need to be checkpointed. As a consequence, PreSync can cause an error at checkpoint when checkpointing initial data with IOUtil::checkpoint_ID. This is because some of these variables are not set during initialization and only during evolution. Since PreSync knows this, it will kill the run when the checkpoint tries to read this invalid data. Therefore, it is critical that thorns properly declare which groups should not be checkpointed to avoid these errors. During normal simulations, PreSync will not complain because checkpoints during evolution have already set all the variables, including RHS and other analysis variables. For checkpointing initial data, though, using the TAGS='checkpoint="no"' is necessary to ensure that thorns behave properly when people use IOUtil::checkpoint_ID to checkpoint after initialization.

Old Thorns

While old thorns can simply adopt the new methodology outright, the reality is that many will likely like to maintain backward compatibility with non-PreSync runs, especially for the near future. Once most thorns are PreSync-capable, full adoption is more viable. In the meantime, many old thorns can adopt a hybrid approach which is PreSync-capable without losing backward compatibility. For most thorns, this should not be a particularly difficult task. The primary difference is in the schedule.ccl. This file simplifies in PreSync, but unfortunately all that goes away with backward compatibility. One must include the old SYNC statements to properly schedule ghost zone synchronization, and the Boundary_Select*ForBCs functions and ApplyBCs groups must be scheduled by hand in the proper places during evolution. The primary code change is for the Boundary_Select*ForBCs functions, as they are replaced by Driver_Select*ForBCs for PreSync. One can either take the path of Kranc and define a Bdy_Select*ForBCs that directly replaces the Boundary_Select*ForBCs. Kranc's functions take the form

static CCTK_INT KrancBdy_SelectVarForBC(
   const CCTK_INT faces,
   const CCTK_INT width,
   const CCTK_INT table_handle,
   const CCTK_STRING var_name,
   const CCTK_STRING bc_name) {

   static bool is_aliased = CCTK_IsFunctionAliased("Driver_SelectGroupForBC");

   int ierr = 0;

   ierr = Boundary_SelectVarForBC(cctkGH_,faces,width,table_handle,var_name,bc_name);

   if(ierr == 0 && is_aliased)
       ierr = Driver_SelectVarForBC(cctkGH_,faces,width,table_handle,var_name,bc_name);

   return ierr;

which always calls Boundary_Select*ForBCs and also calls Driver_Select*ForBCs if it exists. Alternatively, one could leave the Boundary_Select*ForBCs functions alone, make a copy of the BoundarySelect scheduled function, replace the Boundary_Select*ForBCs with Driver_Select*ForBCs in the new scheduled function, and schedule that function in Driver_BoundarySelect as with new thorns.

For more transparency, all of the ApplyBCs calls will immediately exit after their function calls with PreSync on, so these are not very expensive.

  1. TODO: what about SYNC statements? For which choices of presync_mode?

If a thorn does have to use the dynamic read/write functions, these must be surrounded with a

  1. TODO: what if to use? These both start with
assert(not CCTK_EQUALS(presync_mode, "off"));

which will error out if PreSync is turned off.

Extreme Backward Compatibility

The previous section discussed a 'reasonable' level of backward compatibility, but some prefer to have a more rigorous level of backward compatibility. In the case one is mixing new thorns with an old enough flesh, the new macros may not be available. This would require a significantly out-of-date flesh, but for the sake of satisfying all involved, the following preprocessor macro


combined with the function macro


will use the new macros if they exist and otherwise will use the original macro. Most thorn writers will have no need of this; for those that wish to support or intend to use older flesh with newer versions of thorns, this method will work.

Missing topics

This article is not complete. One major piece that needs to be added is the correct way to register a boundary condition with the Boundary thorn using Boundary_RegisterPhysicalBC using the group Boundary_RegisterBCs.