How to Review a new component

From Einstein Toolkit Documentation
Jump to: navigation, search

These notes are currently, as of 2022-08-02, a work in progress.


This document serves as a guideline for reviewers of newly proposed Einstein Toolkit components, it also provides potential contributors with a guideline of what to expect.

Each Einstein Toolkit components has to satisfy a number of criteria for inclusion. These derive from the quality requirements for Einstein Toolkit components laid out in the contributors page, which is authoritative.

The criteria laid out there are:

  • Components should be of sufficient quality to be used for peer-reviewed and published science. This includes a basic standard of software engineering, documentation of the software including algorithms and methods, tutorials and examples, and self-tests to demonstrate that the software works correctly.
  • To focus development and support, optimize resources, and provide best practises, the Einstein Toolkit components should be of current interest to the community.
  • All components must be distributed under an open source licence so that others can use these components without restrictions (except as mandated by scientific integrity), can modify and improve them as necessary, and can pass on these modifications to their collaborators as they see fit.

From these derive guidelines for reviewers.

Formal criteria

These are mostly targeted towards Cactus thorns, but non-thorn contributions should provide equivalent functionality.

  • does the contribution have a README file in the Cactus format? These could be parsed similar to CactusCode's overview page.
    • does it list authors of the component (optional but recommended)?
    • does it list a maintainer who agrees to look after the component?
    • does it list an open source license?
    • does it give a brief description of what the component does?
  • does the contribution have a documentation.tex file that describes its functionality?
    • does it use the Cactus style file and follow the Cactus standards?
    • does it set the "magic markers" % START CACTUS THORNGUIDE and % END CACTUS THORNGUIDE so that it can be included in the online ThornGuide
    • does the documentation build when running make <THORNNAME>-ThornDoc and make <THORNNAME>-ThornDocHTML (the latter requires htlatex)
  • does it have test cases in its tests directory?
  • are there sensible descriptions of all parameters, scheduled routines and grid variables in param.ccl, schedule.ccl and interface.ccl?

Quality criteria

These first two will be handled mostly by the release chair but reviewers are welcome to provide input into these.

  • Has there been a publication using the component? This settles the "sufficient quality" criteria
  • Has there been public interest in the component? For example voiced in the inclusion ticket. This settles the "current interest" criteria

These will be mostly handled by the reviewers

  • Does the code satisfy the criteria laid out in How to Review a Patch
    • is it clear and easy to understand?
    • is it sufficiently commented (using English language comments)?
    • is there "dead code" or are there large commented out sections?
    • are there obvious bugs? Do you notice any corner (or not so corner) cases missing?
  • are externally visible symbols, unless in a C++ namespace or Fortran 90 module in which case the namespace or module name needs to carry the prefix, prefixed with the thorn name? I.e. MyThorn_DoSomthing rather than DoSomething. See Cactus Users Guide for rationale and further details.
  • does the code compile?
  • does it run?
  • does it (obviously) break anything else already in the Einstein Toolkit?
  • do the test cases adhere to Adding a test case
    • do the test cases finish in a reasonable amount of time (less than 10s per test, less than 2min for the whole component)?
    • are the test cases "small" (less than 1GB of memory, preferably much less)
    • do the tests cover important functionality? Science wise? Lines of code wise (using gcov)?

Correctness criteria

Correctness is not part of the review, it having been used in peer reviewed publications is taken as sufficient evidence of correctness.

Optional criteria

A new component reviewer can suggest that the component provides a gallery example. This will have, at least, two advantages:

  1. provide a regularly tested (every 6 months), fully worked out example of the code
  2. increase visibility of new component since it is featured on the Einstein Toolkit home page front page