Einstein Toolkit Documentation - User contributions [en]

ET Working Group Policies and Procedures

2018-06-27T14:05:40Z

Noncct gaballen:

Formalizing the process for members of the consortium to work together to deliver software and tools, provide service, and discuss and brainstorm issues will allow us to

* Coordinate and track efforts from funded projects
* Avoid duplication of efforts
* Improve communication mechanisms around community contributions
* Assist in making processes open and inclusive
* Facilitate proper acknowledgement of contributions
* Facilitate the tracking and collection of metrics around community contributions
* Help encourage contributions from the community

Einstein Toolkit Consortium members are encouraged to initiate and join Development Teams and/or Working Groups:

* Development Teams: Focused on specific software or tool development with clear and documented milestones and deliverables, these teams are anticipated to be short term and transient. Development work is encouraged to take place in an open manner, but may be private until deliverables are delivered.
* Working Groups: These long term groups provide coordination, discuss strategy and opportunities, or support different Einstein Toolkit services are initiatives. Working groups are expected to be open and inclusive and are expected to work through activities such as regular video conferences, workshops, etc.

Processes:

* Working Groups or Development Teams are created by filling out a proposal template (https://docs.einsteintoolkit.org/et-docs/Working_Group_Template) and submitting to maintainers@einsteintoolkit.org. The proposal template will be circulated to ET members and reviewed at an ET call at least one week after circulation. The call should be attended by the proposal lead. In the interim of having a more formal policy, decisions on the proposal will be made by consensus. The review can result in either accepting the proposal, rejecting the proposal, or requesting more information (in which case the review process will be repeated).
* If accepted, the lead will provide a wiki page on the ET web site containing the information in the proposal, and keep the page up to date if any information changes. The lead will also provide monthly updates as requested on the status of activities and progress to milestones and deliverables.
* If no status updates are provided by a group or team for a period of 6 months, the activity will be assumed terminated and will be removed from the ET web page.
* All working group or development team members must be signed up members of the Einstein Toolkit Consortium.

Working List of ET Groups

2017-11-05T17:23:39Z

Noncct gaballen:

# WG: Interface with Experiments (Sasha, Ian H)
##Not just LIGO
##Parameter coordination for folks running simulations
##Making simulations verifiable and reproducible
##Making the code easy to use
##Analysis tools
##Data sharing
##Visualization
#WG: Performance Optimization (Roland, Erik)
##New architectures
##Numerical methods
#WG: Maintainers Club (Steve, Roland, Ian)
##Needs to align with NSF SI2 3.4.1
##Bug fixes
##Releases
##Other things that are done now
#WG: Alternative Theories and Cosmology (Eloisa, Helvi)
##Could be two groups, but need to get things going first
#WG: Matter Codes (Bruno G)
##Some aspects of NSF SI2 3.1.6 and 3.1.5
#WG: Initial Data (Josh Faber, Roland, Eloisa)
##Includes 3.1.2, 3.1.6 from NSF SI2
##Elliptic equations
#WG: Outreach and Education (Gabrielle, Manuela, Helvi)
##NSF SI2 3.4.3?
##Communication
##Industrial relations
##New users
#DG: Datavault (Qian, Gabrielle)
##NSF SI2 3.2 (and see 3.1.2)
#DG: Spherical Coordinate
##NSF SI2 3.1.3
#DG: IllinoisGRMHD (Zach)
##NSF SI2 3.1.5
#DG: Data Dependant Task Scheduler (Steve)
##NSF SI2 3.1.1
##This is already approved and on the wiki

Not included yet
*NSF SI2 3.1.4 - Einstein Exploration Module
*Governance group (NSF SI2 3.4.3, 3.5, etc)

Main Page

2017-11-05T17:18:36Z

Noncct gaballen: /* Einstein Toolkit Working Groups and Development Teams */

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==
{{:meeting_agenda}}

== Einstein Toolkit Working Groups and Development Teams ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved. New working groups should be proposed using the [[Working Group Template]].

'''Planned Groups''':

* [[Working List of ET Groups]]

'''Established Groups:'''

* [[Data Dependant Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [[Tutorial for New Users]]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Working Group Template

2017-11-05T17:17:27Z

Noncct gaballen:

EINSTEIN TOOLKIT WORKING GROUP TEMPLATE

'''Instructions''': To propose a new working group (long term, ongoing activities) or development group (well defined software or tool development with a clear end) fill out this template document and post it to the wiki. (That is, e.g. place it on google doc and put a URL on the wiki for discussion at next weekly meeting).

* Make sure that everyone listed as contributing has agreed to the content.
* Make sure that everyone listed is also listed as a member of the Einstein Toolkit Consortium on our web pages.

'''Template''':

'''Delete as Appropriate''': Working Group / Development Group

'''Title''':

'''Lead(s)''':

'''Initial Members''':

'''Funding''': (e.g. grant details, institutional support, volunteer)

'''Background''': (e.g. explain the motivation, context, and connection to other work)

'''Development''': If a Development Group describe the development activities which will take place

'''Activities''': If a Working Group describe the activities which will be pursued.

'''Milestones''': Include milestones with dates for the group which will be used to track progress.

'''Deliverables''': Include deliverables for the development group or working group with dates (e.g. beta releases, full releases, documentation, workshops, web pages created, etc)

'''Engagement''': Describe how other interested people can get involved or find out more about the activities of the group.

Main Page

2017-11-05T17:13:49Z

Noncct gaballen: /* Einstein Toolkit Working Groups and Development Teams */

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==
{{:meeting_agenda}}

== Einstein Toolkit Working Groups and Development Teams ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved. New working groups should be proposed using the [[Working Group Template]].

* [[Data Dependant Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [[Tutorial for New Users]]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Meeting agenda

2017-10-20T18:07:01Z

Noncct gaballen: /* 2017-10-23 */

<onlyinclude>

The link for the call is: https://hangouts.google.com/hangouts/_/stevenrbrandt.com/etcall. '''After connecting, you will need to wait for the moderator to add you to the call'''.
<includeonly>Today's [[meeting_agenda|meeting agenda]] is on this wiki too.</includeonly>
</onlyinclude>
When adding a topic please add you '''name''' next to the item you are proposing.

== 2017-10-23 ==
* [ES] Introduce Nix https://nixos.org package manager for building Cactus dependencies
* [EB] Discuss the new ET video channel and how to incorporate it in einsteintoolkit.org
* [GA] Help with code optimization from Prace (email from Helvi)

== 2017-10-16 ==

[This will likely overlap with a LIGO/Virgo announcement]

== 2017-10-09 ==

no meeting

== 02-Oct 2017 ==
* [RH] update on Euro ET workshop in Mallorca, Oct 11-14, http://grg.uib.es/EinsteinToolkit2017/
* [RH] thorns for inclusion: GiRaFFE, RNSID. Review status?
* [RH] Jenkins status update
* [RH] Comet lustre data corruption update
* [RH] Next ET release date
Meeting minutes are [http://lists.einsteintoolkit.org/pipermail/users/2017-October/005787.html here].

== 25-Sep 2017 ==

* The cactusmaint@cactuscode.org email address is the one all users are directed to on cactuscode.org, and it seems to not work. Do we want to fix it, or just steer users to users@einsteintoolkit.org?
* Please send to Gabrielle Allen any pictures or movies that can be used for Edfest. A first version of the movie is here: https://www.dropbox.com/s/mpmcwmw6ewdklkf/EdFest.mp4?dl=0
* First working group added to wiki, along with policies, procedures -- suggest that the first meeting of each month includes a review of milestones. I didn't get to work on the others from the SI2 proposal this week :(.
Meeting minutes are [http://lists.einsteintoolkit.org/pipermail/users/2017-September/005780.html here].

== 18-Sep 2017 ==
* Progress on new tutorial machine setup
Meeting minutes are [http://lists.einsteintoolkit.org/pipermail/users/2017-September/005772.html here].

== 11-Sep-2017 ==
* Demo of the RunView tool by Anna Neshyba and David Koppelman. This tool uses PAPI and CactusTimers to generate a graphical output (html) of Cactus performance.
* Is it time for a new release of Cactus? The lack of new releases led at least one online user to conclude that Cactus was a dead project. Perhaps we should have a new Cactus release with each release of the ET?
* Creation of a popular book about the science done by the Einstein Toolkit (Steven) -- I was thinking about ways to improve the visibility of the platform and attract new users and/or funding. Coincidentally, I've recently had some experience figuring out how to publish books on various platforms. These days it's fairly straightforward to publish books with no cost as paperback and ebook, or even hardcover. I was thinking that many of the people working on the ET could write a chapter, and I could handle the publishing. One can configure any level of profit for such endeavors, and I would like to suggest zero, since there are probably legal issues with the toolkit actually making money from such a thing. Does this sound like something people would be interested in?
* switch ExternalLibraries over to github: svn checkout https://github.com/rhaas80/GSL

Meeting minutes are [http://lists.einsteintoolkit.org/pipermail/users/2017-September/005769.html here].

== 28-Aug-2017 ==
* Status update on KNL
* update on Euro ET workshop in Mallorca, Oct 11-14, http://grg.uib.es/EinsteinToolkit2017/
* status of servers at LSU
** continued use of svn
** switch ExternalLibraries over to github: svn checkout https://github.com/rhaas80/GSL
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005739.html

== 21-Aug-2017 ==
* Progress on tutorial machine at NCSA
* Comments/discussion on updates to the Working Group Document (including procedures) (Gabrielle) https://docs.google.com/document/d/1j12IEY8kH6R33qfXpD5BaDDXLEYdNceEqf9crLupsPs
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005739.html

== 14-Aug-2017 ==
* Report on ET workshop survey (Nasir Eisty)
* NSF SI2 Project posted (Gabrielle Allen) https://figshare.com/articles/Einstein_Toolkit_Community_Integration_and_Data_Exploration/5306587
* Template for working groups: https://docs.google.com/document/d/1j12IEY8kH6R33qfXpD5BaDDXLEYdNceEqf9crLupsPs/edit?usp=sharing
* Discuss new code contributions
** RNSID

== 07-Aug-2017 ==

* Report from summer school and workshop:
** Summer school recordings etc. are here: https://docs.einsteintoolkit.org/et-docs/NCSAETK2017/setup
* Introduce working groups
* Discuss new code contributions
** GiRAFFE
** RNSID
* DataVault updates
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005706.html

Meeting agenda

2017-09-25T14:02:59Z

Noncct gaballen: /* 25-Sep 2017 */

<onlyinclude>

The link for the call is: https://hangouts.google.com/hangouts/_/stevenrbrandt.com/etcall. '''After connecting, you will need to wait for the moderator to add you to the call'''.
<includeonly>Today's [[meeting_agenda|meeting agenda]] is on this wiki too.</includeonly>
</onlyinclude>
When adding a topic please add you '''name''' next to the item you are proposing.

== 25-Sep 2017 ==

* The cactusmaint@cactuscode.org email address is the one all users are directed to on cactuscode.org, and it seems to not work. Do we want to fix it, or just steer users to users@einsteintoolkit.org?
* Please send to Gabrielle Allen any pictures or movies that can be used for Edfest. A first version of the movie is here: https://www.dropbox.com/s/mpmcwmw6ewdklkf/EdFest.mp4?dl=0
* First working group added to wiki, along with policies, procedures -- suggest that the first meeting of each month includes a review of milestones. I didn't get to work on the others from the SI2 proposal this week :(.

== 18-Sep 2017 ==
* Progress on new tutorial machine setup

== 11-Sep-2017 ==
* Demo of the RunView tool by Anna Neshyba and David Koppelman. This tool uses PAPI and CactusTimers to generate a graphical output (html) of Cactus performance.
* Is it time for a new release of Cactus? The lack of new releases led at least one online user to conclude that Cactus was a dead project. Perhaps we should have a new Cactus release with each release of the ET?
* Creation of a popular book about the science done by the Einstein Toolkit (Steven) -- I was thinking about ways to improve the visibility of the platform and attract new users and/or funding. Coincidentally, I've recently had some experience figuring out how to publish books on various platforms. These days it's fairly straightforward to publish books with no cost as paperback and ebook, or even hardcover. I was thinking that many of the people working on the ET could write a chapter, and I could handle the publishing. One can configure any level of profit for such endeavors, and I would like to suggest zero, since there are probably legal issues with the toolkit actually making money from such a thing. Does this sound like something people would be interested in?
* switch ExternalLibraries over to github: svn checkout https://github.com/rhaas80/GSL

Meeting minutes are [http://lists.einsteintoolkit.org/pipermail/users/2017-September/005769.html here].

== 28-Aug-2017 ==
* Status update on KNL
* update on Euro ET workshop in Mallorca, Oct 11-14, http://grg.uib.es/EinsteinToolkit2017/
* status of servers at LSU
** continued use of svn
** switch ExternalLibraries over to github: svn checkout https://github.com/rhaas80/GSL
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005739.html

== 21-Aug-2017 ==
* Progress on tutorial machine at NCSA
* Comments/discussion on updates to the Working Group Document (including procedures) (Gabrielle) https://docs.google.com/document/d/1j12IEY8kH6R33qfXpD5BaDDXLEYdNceEqf9crLupsPs
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005739.html

== 14-Aug-2017 ==
* Report on ET workshop survey (Nasir Eisty)
* NSF SI2 Project posted (Gabrielle Allen) https://figshare.com/articles/Einstein_Toolkit_Community_Integration_and_Data_Exploration/5306587
* Template for working groups: https://docs.google.com/document/d/1j12IEY8kH6R33qfXpD5BaDDXLEYdNceEqf9crLupsPs/edit?usp=sharing
* Discuss new code contributions
** RNSID

== 07-Aug-2017 ==

* Report from summer school and workshop:
** Summer school recordings etc. are here: https://docs.einsteintoolkit.org/et-docs/NCSAETK2017/setup
* Introduce working groups
* Discuss new code contributions
** GiRAFFE
** RNSID
* DataVault updates
Minutes: http://lists.einsteintoolkit.org/pipermail/users/2017-August/005706.html

Data Dependant Task Scheduler

2017-09-25T13:58:46Z

Noncct gaballen:

'''Development Team: Data Dependant Task Scheduler'''

'''Date Approved:''' August 13th, 2017

'''Lead:''' Steve Brandt

'''Team:''' Roland Haas, Erik Schnetter, Frank Loeffler, Samuel Cupp

'''Funding:''' NSF 1550514, NSF 1550551, Perimeter Institute

'''Background:'''

Writing code with the Einstein Toolkit is becoming more challenging because of the increase in the need to handle highly complex multi-physics, as well as the increase in parallelism and inhomogeneity of even desktop-size hardware. At the heart of every simulation is orchestrating the task implied by the different science and computational components, making sure that available computational and memory resources are used efficiently. And this is the job of the scheduler in the Einstein Toolkit. The scheduler is provided by Cactus. It provides time bins for starting up, evolving a step, and shutting down. Cactus programming modules (i.e. thorns) can schedule functions inside any of these bins, insert new scheduling bins inside this basic skeleton, and arrange them before or after others. Thorns modify this schedule through the use of a domain specific language (DSL). Halo exchanges for variables, or groups of variables, can be triggered by annotating these schedule bins with a sync directive. Identifying the correct places to perform synchronization has proven difficult in many cases. This have sometimes resulted in bugs, but more often in unnecessary synchronization leading to performance loss. This problem also manifests inside the adaptive mesh refinement component of the Einstein Toolkit. While initial applications followed the time-stepping requirements of the overall Berger and Oliger method, some quantities needed to be evaluated in ways that did not fit into this structure, like global reductions for analysis purposes, or multigrid cycles. Special scheduling annotations were created to facilitate the needs of these calculations. Science was served, but the programmability and overall usability of the Cactus Framework suffered.

On execution, Cactus traverses the schedule tree in order and tells the driver to execute the routines scheduled, by the active thorns, in the order requested by the thorn writers and communicate ghost zone data when necessary. Every requested data synchronization constitutes an explicit barrier, as further execution does not continue before the communication has completed. This is the case, even if the next scheduled routine does not depend on the data being communicated after the previous routine. Thus, it is currently not possible to interleave computation and communication and the explicit barriers makes it impossible for a part of the grid to get ahead of another part of the grid. In order to improve this situation, the present scheduler needs to be replaced with a much more flexible approach, relying explicitly on data dependencies.

Other libraries and frameworks of relevance include Charm++, HPX, FunHPC, SpECTRE (which uses Charm++, Jonah Miller wrote the first library to use Charm++ for SpECTRE), ExaHype, and GAMER (fully GPU/AMR library at UIUC written by Justin Shrive).

'''Development:'''

We will modify the flesh of Cactus to allow a thorn writer to annotate the schedule description with information about which variables a routine reads and writes as an alternative to the existing method of specifying which functions run before or after each other, and which are to be synchronized. This information will be used to construct a dependency graph using standard C++ futures.

The main wiki for development of the standard as well as current status of the project and working group may be found here:

https://docs.einsteintoolkit.org/et-docs/Adding_requirements_to_the_Cactus_scheduler

'''Milestones:'''

* October 2017: Identify methods to correctly specify READS and WRITES clauses in Cactus thorns, including procedures to modify existing thorns and make them compliant. In addition, modify the behavior of synchronization so that SYNC statements are no longer necessary.
* January 2018: Replace the static scheduler with a data-dependent scheduler, and adapt PUGH to work with this scheduler. Begin usage of C++ futures for parallelism.
* September 2018: Adapt Carpet to the new data-dependent scheduler with a simpler, but working PUGH reference implementation.
* September 2019: Complete adapting Carpet internals to use the new scheduler. Hardening, testing, and performance optimization. Adapt key thorns to make better use of the new infrastructure. Cactus user’s workshop to demonstrate the new scheduler.
* September 2020: Continue adaptation of science modules. Development of an online tutorial.

'''Deliverables:'''

September 2018: PUGH Reference Implementation Released

September 2019: New scheduler available in regular release in Carpet

September 2020: Online tutorial.

'''Engagement:'''

If you wish to contribute, please add your name to the working group on
https://docs.einsteintoolkit.org/et-docs/Adding_requirements_to_the_Cactus_scheduler and contact Steven R. Brandt (sbrandt@cct.lsu.edu) to let him know your interest.

Main Page

2017-09-25T13:53:04Z

Noncct gaballen: /* Einstein Toolkit Working Groups and Development Teams */

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==
{{:meeting_agenda}}

== Einstein Toolkit Working Groups and Development Teams ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved.

* [[Data Dependant Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [[Tutorial for New Users]]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

Main Page

2017-09-25T13:52:31Z

Noncct gaballen: /* Einstein Toolkit Working Groups */

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==
{{:meeting_agenda}}

== Einstein Toolkit Working Groups and Development Teams ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved.

* [[DT Data Dependant Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [[Tutorial for New Users]]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]

ET Working Group Policies and Procedures

2017-09-25T13:51:25Z

Noncct gaballen:

Formalizing the process for members of the consortium to work together to deliver software and tools, provide service, and discuss and brainstorm issues will allow us to

* Coordinate and track efforts from funded projects
* Avoid duplication of efforts
* Improve communication mechanisms around community contributions
* Assist in making processes open and inclusive
* Facilitate proper acknowledgement of contributions
* Facilitate the tracking and collection of metrics around community contributions
* Help encourage contributions from the community

Einstein Toolkit Consortium members are encouraged to initiate and join Development Teams and/or Working Groups:

* Development Teams: Focused on specific software or tool development with clear and documented milestones and deliverables, these teams are anticipated to be short term and transient. Development work is encouraged to take place in an open manner, but may be private until deliverables are delivered.
* Working Groups: These long term groups provide coordination, discuss strategy and opportunities, or support different Einstein Toolkit services are initiatives. Working groups are expected to be open and inclusive and are expected to work through activities such as regular video conferences, workshops, etc.

Processes:

* Working Groups or Development Teams are created by filling out a proposal template (give link) and submitting to maintainers@einsteintoolkit.org. The proposal template will be circulated to ET members and reviewed at an ET call at least one week after circulation. The call should be attended by the proposal lead. In the interim of having a more formal policy, decisions on the proposal will be made by consensus. The review can result in either accepting the proposal, rejecting the proposal, or requesting more information (in which case the review process will be repeated).
* If accepted, the lead will provide a wiki page on the ET web site containing the information in the proposal, and keep the page up to date if any information changes. The lead will also provide monthly updates as requested on the status of activities and progress to milestones and deliverables.
* If no status updates are provided by a group or team for a period of 6 months, the activity will be assumed terminated and will be removed from the ET web page.
* All working group or development team members must be signed up members of the Einstein Toolkit Consortium.

Main Page

2017-09-25T13:49:42Z

Noncct gaballen:

==Instructions==

Documentation for the [http://www.einsteintoolkit.org Einstein Toolkit] is a community effort and everyone is encouraged to contribute towards these pages. You need to login to the wiki to edit the documentation, to do this either

* if you already have a personal LDAP login with the CCT use this
* create you own login/password for the wiki (select local domain)

If you would like to make major changes to these pages please discuss first on the users@einsteintoolkit.org mail list.

Thanks for your contributions!

== Weekly Users Call ==
{{:meeting_agenda}}

== Einstein Toolkit Working Groups ==

The idea of establishing working groups for the Einstein Toolkit was discussed at the Summer 2017 Workshop at NCSA, and is now being implemented. The initial [[ET Working Group Policies and Procedures]] document has been discussed and approved.

* [[WG Data Dependant Task Scheduler]]

==Einstein Toolkit workshop information==

=== Current ===
* [http://grg.uib.es/EinsteinToolkit2017/ ET Workshop Fall 2017 and EdFest, Mallorca]

=== Past ===

* [[ET Workshop 2017 at NCSA| ET Workshop Summer 2017, NCSA]]
* [[2017 MHD Workshop| (non-ET) MHD workshop, 2017]]
* [[ET Workshop 2016| ET Workshop Summer 2016, Trento]]
* [[ET Workshop 2015| ET Workshop Summer 2015, Stockholm]]
* [[ET_Workshop Summer 2013]] (shows both workshops)
** [[ET_Workshop Summer 2013 (New Users Workshop)]]
** [[ET_Workshop Summer 2013 (Developers Workshop)]]
* [[ET_Workshop Fall 2012]]
* [[ET_Workshop Spring 2012]]
* [[ET_Workshop Fall 2011]]
* [[ET_Workshop Spring 2011]]
* [http://ccrg.rit.edu/~carpet/index.php/Main_Page Carpet Developer Workshop Summer 2010]

==Einstein Toolkit Seminars==

It would be nice to organize a semi-regular series of talks with topics of general interest. This should include
not only people from within the Einstein Toolkit. The ET web page already contains a (currently quite short)
[http://einsteintoolkit.org/seminars/ list], and now it is time to propose upcoming talks. In order to record and
story voiced proposals, please use the wiki page [[Einstein Toolkit Seminar Proposals]].

==Release planning==
* [[Release Details]] - Plan, status and checklist for the upcoming release
* [[Release coordination]] - Discussion of the upcoming release
* [[Release Process]] - Description of the process that we go through for each release

==Documentation==

* [[Tutorial for New Users]]
* [[Simplified Tutorial for New Users]] (requires Debian, Linux Mint, Fedora or Mac OSX)
* [[Getting Started for Cactus Experts]]
* [[Thorns_we_know_of|Non-Einstein-Toolkit thorns]]
* [[Einstein Toolkit standards]]: ADMBase, HydroBase, SphericalSurface
* [[Supported Machines]], [[Configuring a new machine]]
* [[Machines]] - Notes for specific machines
* [[Using Eclipse / Mojave]]
* [[Running Cactus On Windows]]
* Adding your own analysis method
* Adding your own [[adding initial data|initial data]]
* Adding a [[adding a test case|test case]]
* CCE [http://ccrg.rit.edu/~yosef/cce.html tutorial]
* [[Visualizing magnetic field lines]]
* [[Using the multi-model mechanism in Carpet]]
* [[Working with git]]
* [[FAQ]]
* [[Compiling the Einstein Toolkit]]
* [[Analysis and post-processing]]
* [[Editing the Einstein Toolkit website]]
* [[Licences]]

==Regression Test Results==
You can run [[Simulation_Factory_Advanced_Tutorial#Test_suites | regression tests using SimFactory]].

Results from automated tests are available at https://build.barrywardell.net.

==Performance Test Results==
* [[Single-node benchmark results]] for the Einstein Toolkit
* TODO: Scalability benchmark results for the Einstein Toolkit

==Projects==

* [[Adding requirements to the Cactus scheduler]]
* [[Visualization of simulation results]]
* [[Automated testing]]
* [[Vectorization]] Improving code performance by using the CPU's vector instructions
* [[Padding]] Improving code performance by optimizing cache access
* [[Making Cactus Installable]] like any other Ubuntu package
* [[Test suite results are unwieldy]]
* [[Summer student projects]]
* [[Improving the treatment of external libraries]]
* [[Piraha Parser Discussion]]
* [[Version control]]
* [[Rewrite McLachlan]]
* [[Fixing examples]]
* [[A new I/O file format]]
* [[Adding Llama to the ET]]
* [[Remote Mini-Workshop Series]]
* [[Running Cactus on Knights Landing]]
* [[Improving the new user experience]]

==Maintainers==

* [[Organization and Responsibilities]]
* [[How to Review a Patch]]
* [[Policies to retire functionality]]
* [[Preparing a Patch for Review]]
* [http://einsteintoolkit.org/release-info/parse_testsuite_results.php Release Testsuite Status], [https://build.barrywardell.net/job/EinsteinToolkit/ Trunk Testsuite Status]
* [[Standard Emails]]
* [[MHD implementation details and discussions]]
* [[Carpet Wish List]]
* [[Editing the website]]
* [[Usage poll]]
* [[Repository transition]]
* [[Tickets]]

== Getting Started with Wikis ==

* Consult the [http://meta.wikimedia.org/wiki/Help:Contents User's Guide] for information on using the wiki software.
* [http://www.mediawiki.org/wiki/Help:Configuration_settings Configuration settings list]
* [http://www.mediawiki.org/wiki/Help:FAQ MediaWiki FAQ]
* [http://mail.wikimedia.org/mailman/listinfo/mediawiki-announce MediaWiki release mailing list]