Adding requirements to the Cactus scheduler - Revision history

Sbrandt at 15:29, 16 August 2017

2017-08-16T15:29:51Z

Sbrandt at 15:28, 16 August 2017

2017-08-16T15:28:39Z

Sbrandt at 15:26, 16 August 2017

2017-08-16T15:26:28Z

130.39.94.46: /* Current State */

2015-04-28T13:26:01Z

‎Current State

130.39.94.46: /* Current State */

2015-04-28T13:12:06Z

‎Current State

130.39.94.46: /* Notes based on a Telecon dated 2/21/2013 */

2015-04-28T13:09:33Z

‎Notes based on a Telecon dated 2/21/2013

130.39.94.46: /* Next Steps */

2015-04-28T13:08:10Z

‎Next Steps

130.39.131.136: /* Suggested Solution */

2013-02-21T13:04:51Z

‎Suggested Solution

194.94.224.254 at 09:05, 14 August 2012

2012-08-14T09:05:55Z

Hinder: /* Suggested Solution */

2011-11-02T09:56:00Z

‎Suggested Solution

←Older revision		Revision as of 15:29, 16 August 2017
Line 1:		Line 1:
		+	==Working Group==
		+
		+	* Samuel Cupp
		+	* Steven R. Brandt
		+	* Erik Schnetter
		+
		+	If you want to contribute, please add your name to the list above and email Steven R. Brandt at sbrandt@cct.lsu.edu.
		+
	==Problem Outline==		==Problem Outline==

←Older revision		Revision as of 15:28, 16 August 2017
Line 46:		Line 46:

	==Current State==		==Current State==
		+
		+	Current effort at LSU: https://github.com/stevenrbrandt/PresyncWave.git
		+
		+	This effort does only a part of the task, automating the synchronization, application of boundary conditions, and ensuring that the read and write directives are present and correct. Currently, LSU is working on getting the system to function for the static_tov.par example.

	The flesh that allows adding REQUIRES and PROVIDES clauses to the schedule block for every routine. They are stored in the schedule database of the flesh and are ignored by default. There is a suggestion to rename these clauses to READS and WRITES; this has not yet been done.		The flesh that allows adding REQUIRES and PROVIDES clauses to the schedule block for every routine. They are stored in the schedule database of the flesh and are ignored by default. There is a suggestion to rename these clauses to READS and WRITES; this has not yet been done.

@@ Line 7: / Line 7: @@
 The Cactus schedule offers "before" and "after" clauses to ensure a partial ordering between routines. Unfortunately, this ordering applies only to routines within the same schedule group and the same schedule bin and refinement level. It is not possible to ensure a particular order between routines in different schedule groups or schedule bins, and it is very complex to ensure that a routine is executed e.g. after another routine has been executed on all refinement levels.
-There is one example setup that illustrates this problem. When setting up initial conditions for a hydrodynamics evolution, one may e.g. want to first set up a neutron star, then calculate its maximum density, and then set up the atmosphere to a value depending on this maximum density. Making this possible in Cactus required introducing a new schedule bin "postpostinitial" to the flesh, and requires careful arrangement of schedule groups defined by ADMBase and HydroBase. Even now that this is possible, it is probably not possible to ensure at run time that these actions occur in a correct order.
+There is one example setup that illustrates this problem. When setting up initial conditions for a hydrodynamics evolution, one may e.g. want to first set up a neutron star, then calculate its maximum density, and then set up the atmosphere to a value depending on this maximum density. Making this possible in Cactus required introducing a new schedule bin "postpostinitial" to the flesh, and requires careful arrangement of schedule groups defined by ADMBase and HydroBase. Even now that this is possible, it is probably not possible to ensure at run time that these actions occur in a correct order. (Note: Get a more precise definition of this use case so we can implement and test the solution)
 ==Suggested Solution==
@@ Line 15: / Line 15: @@
 * Each scheduled routine declares which grid variables it reads and which grid variables it writes
 * Since most routines write only parts of grid variables, the routine would also specify which part it reads/writes, e.g. the interior, outer boundary, symmetry boundary, etc.
 * This allows the Cactus scheduler in a first step to validate the schedule and detect cases where a required variable has not been defined, or where a variable is calculated multiple times or synchronized multiple times
 * In a second step this will also allow the Cactus scheduler to completely derive the schedule from these declarations. This may even make it possible to execute routines in parallel if they are independent. Even SYNC statements can be automatically derived, and schedule groups would not be necessary any more.

←Older revision		Revision as of 13:26, 28 April 2015
Line 59:		Line 59:

	Directives which omit the domain (i.e. the value ([Interior\|Boundary\|Everywhere]) are assumed to mean "Everywhere."		Directives which omit the domain (i.e. the value ([Interior\|Boundary\|Everywhere]) are assumed to mean "Everywhere."
		+
		+	Steven Brandt has developed a scheduler modification which uses reads and writes clauses to generate a partial order within a group. If there are multiple writes to the same variable or a dependency cycle exists it is a fatal error, e.g.
		+
		+	schedule foo1 in MoL_CalcRHS
		+	{
		+	LANG: C
		+	WRITES: example::a
		+	READS: example::b
		+	}
		+	schedule foo2 in MoL_CalcRHS
		+	{
		+	LANG: C
		+	WRITES: example::b
		+	READS: example::a
		+	}
		+
		+	This situation can be disambiguated with a version string, e.g.
		+
		+	schedule foo1 in MoL_CalcRHS
		+	{
		+	LANG: C
		+	WRITES: example::a$v2
		+	READS: example::b
		+	}
		+	schedule foo2 in MoL_CalcRHS
		+	{
		+	LANG: C
		+	WRITES: example::b
		+	READS: example::a
		+	}
		+
		+	Although the names example::a$v2 and example::a refer to the same grid function, the scheduler does not see an ordering relationship between them. Thus, in the above example, foo2 executes first because foo1 depends on it as a result of example::b.
		+
		+	Ordering of schedule items is done per group, as before, however, groups inherit reads/writes clauses from their sub-items.
		+
		+	There are clear deficiencies in this scheme. It does not address the problems related to AMR outlined in the statement of the problem. The solution does, however, allow the reads and writes clauses generated by Kranc to have an effect. Is this a step in the right direction?

	==Next Steps==		==Next Steps==

←Older revision		Revision as of 13:12, 28 April 2015
Line 53:		Line 53:

	Carpet has a file Requirements.cc that detects the presence of these clauses and performs rudimentary checks. These checks are probably useless in their current form.		Carpet has a file Requirements.cc that detects the presence of these clauses and performs rudimentary checks. These checks are probably useless in their current form.
		+
		+	UPDATE: The CCL parser currently parses reads/writes directives of the form:
		+
		+	IMPLEMENTATION_NAME::GF_NAME([Interior\|Boundary\|Everywhere])
		+
		+	Directives which omit the domain (i.e. the value ([Interior\|Boundary\|Everywhere]) are assumed to mean "Everywhere."

	==Next Steps==		==Next Steps==

@@ Line 36: / Line 36: @@
+  }
-This solution ensures appears to be the easiest way to get the benefits of compile-time checking (e.g. const and intent(in) declarations, etc.), and keeping the code and schedule in sync with each other. The schedule.ccl should also be updated to support multiple "at" or "in" declarations for a single function.
+This solution appears to be the easiest way to get the benefits of compile-time checking (e.g. const and intent(in) declarations, etc.), and keeping the code and schedule in sync with each other. The schedule.ccl should also be updated to support multiple "at" or "in" declarations for a single function.
 We also agreed, however, that we need a dynamic mechanism for special kinds of functions (I/O routines, for example).

@@ Line 56: / Line 56: @@
 ==Next Steps==
-To bring this project further, we need to define how the "reads" and "writes" clauses should look like. As mentioned above, it is insufficient to list only grid variables there, since most routines access only parts of grid variables. '''Ian Hinder''' volunteered to come up with an initial plan for what kinds of "parts" there should be (e.g. interior, outer boundary, symmetry boundary, ghost zone, etc.). Those parts are driver-dependent, which means we have to come up with a way to tell the flesh about those parts and their connections (what is part of what).
+To bring this project further, we need to define what the "reads" and "writes" clauses should look like. As mentioned above, it is insufficient to list only grid variables there, since most routines access only parts of grid variables. '''Ian Hinder''' volunteered to come up with an initial plan for what kinds of "parts" there should be (e.g. interior, outer boundary, symmetry boundary, ghost zone, etc.). Those parts are driver-dependent, which means we have to come up with a way to tell the flesh about those parts and their connections (what is part of what).
 A simple example would look like the following (syntax arbitrary):

←Older revision		Revision as of 13:04, 21 February 2013
Line 25:		Line 25:

	Another issue arises with loops in the schedule. This is currently mostly used by MoL for the sub-timesteps. There is currently no good idea for handling this. Logically those loops can be seen as a nested schedule tree: it should be possible to do the same as for the complete tree.		Another issue arises with loops in the schedule. This is currently mostly used by MoL for the sub-timesteps. There is currently no good idea for handling this. Logically those loops can be seen as a nested schedule tree: it should be possible to do the same as for the complete tree.
		+
		+	==Notes based on a Telecon dated 2/21/2013==
		+
		+	The solution we think we've converged on is to put reads and writes declarations in the schedule tree, and then generate function-specific macros, e.g.
		+
		+	void funwave_dispersion(CCTK_ARGUMENTS)
		+	{
		+	DECLARE_CCTK_ARGUMENTS_funwave_dispersion;
		+	...
		+	}
		+
		+	This solution ensures appears to be the easiest way to get the benefits of compile-time checking (e.g. const and intent(in) declarations, etc.), and keeping the code and schedule in sync with each other. The schedule.ccl should also be updated to support multiple "at" or "in" declarations for a single function.
		+
		+	We also agreed, however, that we need a dynamic mechanism for special kinds of functions (I/O routines, for example).
		+
		+	With regard to fine-tuning of where functions get read or written, we want to only worry about interior or boundary/ghost zones at the moment. Something simple. There is the possibility of calling a user function on only a subsection of the grid. This is a subject of ongoing discussion.
		+
		+	We want to plan a video workshop on this topic for mid-march (2 hours) with some warmups. If we can get things working properly, this sort of workshop can occur more regularly.

	==Current State==		==Current State==

←Older revision		Revision as of 09:05, 14 August 2012
Line 198:		Line 198:

	* Investigate workflow systems, e.g. as DAGMan, GWES, Triana, or ProC.		* Investigate workflow systems, e.g. as DAGMan, GWES, Triana, or ProC.
		+
		+	* If it is known at CST-time which variables are only read and not written by a scheduled function, the CST could generate const pointers for those variables to ensure that the data is not written to.

@@ Line 22: / Line 22: @@
 One possibility to resolve this would be to add a tag to variables, declaring that this routine "reads Aij:original" and writes "Aij:constraints-enforced". Each other routine accessing this variables would then also need to declare whether it reads or writes the original Aij or the Aij with constraints enforced. This is also the problem of this mechanism: it would create unwanted thorn-dependencies.
-Another possibility would be to use the existing BEFORE/AFTER mechanism for those cases where it is generally not possible to define proper data dependencies through variables alone. This would make it very easy to add a function which modifies e.g. Aij easily at any place, without making other thorns depend on the presence of this inserted routine.
+Another possibility would be to use the existing BEFORE/AFTER mechanism for those cases where it is generally not possible to define proper data dependencies through variables alone. This would make it very easy to add a function which modifies e.g. Aij easily at any place, without making other thorns depend on the presence of this inserted routine.  ICH: As a result, we think the BEFORE/AFTER mechanism will be necessary, and it is not worth developing the tagging/aliasing mechanism at this point.
 Another issue arises with loops in the schedule. This is currently mostly used by MoL for the sub-timesteps. There is currently no good idea for handling this. Logically those loops can be seen as a nested schedule tree: it should be possible to do the same as for the complete tree.