Release Process

From Einstein Toolkit Documentation
Jump to: navigation, search

Release Process

This page describes the release process in some detail. Getting a release out of the door without embarrassing oversights requires a bit more planning and care than one would like.

Timeline for a release and estimated time required

  1. Form a release coordination team (6 months) (part of ET calls)
    1. a release coordinator
    2. a release coordinator assistant, who may be the release coordinator for the next release
    3. the former release coordinator to help out
    4. test runner(s) who will run the testsuites on the list of supported machines
    5. gallery runners for gallery examples
    6. final list may include others contributing directly to the release process
  2. Come up with release timeline (by translating the last release timeline forward in time), see (5 months) (30 minutes), see
    1. Release coordinator needs write access to the following repos (or must arrange all needed changes with a repo maintainer)
    2. Frequently remind participants of timeline (part of ET calls)
    3. Begin discussions on the mailing list, reminding people to look at test cases, review patches etc. (5 minutes per week)
  3. Choose features which are going to be included, and those which won't be included (4 months)
    1. ask for review volunteers, assign tasks to people (part of ET calls) e.g.
    2. create tickets for each proposed item on eg.
    3. Decide on a list of release-critical compute systems eg XSEDE, LCCF, DOE, PRACE:
      1. get accounts for test runners
  4. Drive review of thorns for code quality, documentation, test cases (continuously)
    1. Include positively reviewed and voted on items in main ET repositories
    2. Remove new features without reviewer or negatively reviewed from list of new features
    3. Frequently remind participants of timeline
  5. Start testing on release-critical compute systems (2 months) (30 minutes per week per cluster to compile, run, check, upload)
    1. initially test to make sure clusters can still build code
    2. test code as they become positively reviewed (code is present in master before final positive review)
    3. report on test failures
    4. assign found bugs to volunteers
  6. Release preparation I (4 weeks)
    1. Announce feature freeze based on currently positively reviewed components
      1. fix branch to use for kuibit to one of the tagged releases. TODO: add instructions on how to use that branch when creating ET branches
    2. Collect list of new features, newsworthy items, and acknowledgements for release announcement on wiki (there’s a script in the wiki to help with this) (45 minutes)
    3. Create Zenodo item (notes) (1 hour)
      1. Send to for review re typos etc.
      2. Include in "do you want to be included" email to contributors.
    4. Contact contributors and release team about permanent (contributors) and one-off (release team) inclusion in the ET author list (10 minutes of work per author to poke and prod them to respond). Include in cc to have a record.
    5. announce final member list of release team:
  • release coordinator(s)
  • test runner(s)
  • gallery example runner(s)
  • reviewer(s)
  • anyone putting in work specifically for the release
    1. Draft release announcement (1 hour)
      1. send to for review re typos etc.
    2. Announce release date publicly
  1. Release preparation II (1 week), detailed list of items and partial scripts at end of text
    1. Finalize announcement draft (1 hour)
      1. Send to for review re typos etc.
    2. Update einsteintoolkit.bib ( with
      1. Update or add requested and suggested citations for all ET components in thorn list (10 minutes).
      2. Latest release info and DOI (from Zenodo entry above)
      3. move requested-for for main ET cite (Zenodo)
    3. Update version information in Makefile, the documentation tex files and Baikal documentation.tex file script (15 minutes)
    4. Regenerate all auto-generated files (30 minutes of work, plus time to wait for Kranc)
    5. Create release branch and tag for all ET codes (30 minutes)
    6. Update websites in a branch (2 hours)
      1. ( ; be sure to update the main page; then about/releases/; then from that directory run `` [you'll need to `pip install markdown` for this to work]; then be sure to `git add` and commit the new files.)
      2. (, see below for a script.
      3. ( make life using
      4. CactusTutorial.ipynb (
      5. update docker image for tutorial server script, but do not yet push
  2. The release (0 days)
    1. make all updated branches live (30 min)
    2. push tutorial server docker image to Dockerhub (only Erik, Steve, Roland, Ian can do so)
    3. announce on, (5 min)
    4. based on previous release announcement (15 min)
    5. update Wikipedia Cactus entry (5 min)
    6. Run the testsuite using the release code on all supported machines (30 min per cluster)
    7. Update with lessons learned (30 minutes)

Helpful tools and notes

This section collects a handful of useful commands that may become critical to guarantee the quality of the release. The descriptions below are quite technical and intended for the developers and maintainers only.

add gallery runners

Gallery runners should be mentioned in their respective gallery example's ticket and should have the ticket assigned to them. This requires that they are a member of the Einstein Toolkit bitbucket organization.

  • they must be invited to it using their email address on:
  • they should be made members of the the "All Members" and "gallery runners" groups. The former to be assigned the ticket, the latter to be able to upload files to the website repository and edit the web-pages
  • find the gallery ticket and use the "Edit" button to assign users
  • gallery examples should be run using the "master" branch of the toolkit, that will become the release branch, of manifest and thornlist
  • results should be archived in a tar archive as shown on and the archive name should include year, month and day of month YYYYMMDD following the pattern visible in the existing tar files. On macOS make sure to not include __MACOSX a folder. This is most easily achieved by using the command line zip tool and not a GUI tool or Finder (to be verified)
  • some archives are large and special tools are needed to upload them, this is described in the ticket for the example
  • after updating the archive, clone the website repo, edit the gallery page to point to the new archive file and update the "last run" line. Don't forget to update and add any images and plots you have regenerated. Verify the change by opening the html file in a local browsers. Commit and push to nmaster.

update developers.txt

Inside , you will find a developers.txt file, which needs to be updated with any new developers for this release. Any developers who helped with the last release but have not yet contributed code to the Toolkit need to be removed, and likewise any new folks who contributed to this release need to be added.

There is a script below under "Other Stuff" for getting the contributors from the git repos.

After updating, run

Next, update It contains the hard-coded information in between the EDIT BELOW and EDIT ABOVE tags.

 release_team = [
   "Zachariah B. Etienne",
   "Roland Haas",
   "Steven R. Brandt",
   "William E. Gabella",
   "Peter Diener",
   "Atul Kedia",
   "Miguel Gracia",
 contributers = [
   "Roland Haas",
   "Steven R. Brandt",
 et_release = "ET_2021_05"
 et_release_codename = "Lorentz"
 et_description = ...

Publish to the sandbox (following the steps below). Once that is done and reviewed, it can be published without --sandbox.

drafting Zenodo entry

# get Steve Brandt's personal access token for
# create your own personal access token for

# note down ID of "Einstein Toolkit" (will change each release)
ZENODO_ID=$(python3 ./ --list | perl -n -e '/(\d*) The Einstein Toolkit/ && print ($1) && exit 0')

# This will generate and (metadata)
python3 ./ --id $ZENODO_ID --generate

# recreate current release in sandbox, note that will use
# SANDBOX_ZENODO_ACCESS token and when --sandbox is used.
# note down ID of the deposit
ZENODO_SANDBOX_ID=$(python3 ./ --sandbox --create | perl -n -e '/(\d*) The Einstein Toolkit/ && print ($1) && exit 0')

# publish current release in sandbox
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --deposit
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --publish

# now we make the new release

# Edit the lines in that are between the Edit Below/Edit Above comments.
# This includes the release, the code name, the release team, etc.

# This will generate and (metadata)
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --generate

# create a new version in sandbox
# and note down its id
ZENODO_SANDBOX_ID=$(python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --newversion | perl -n -e '/(\d*) The Einstein Toolkit/ && print ($1) && exit 0')

# upload updated metadata (
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --upload

# deposit updated dummy file
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --deposit
# publish new version (cannot be undone)
python3 ./ --sandbox --id $ZENODO_SANDBOX_ID --publish

# create a new version in real Zenodo but *do not* publish (yet)
# and not down its id
ZENODO_ID=$(python3 ./ --id $ZENODO_ID --newversion)

echo "Zenodo ID is: $ZENODO_ID."
echo "You MUST write down this ID since I will not be able to retrieve it later"

# generate metadata for actual Zenodo (different DOI!)
python3 ./ --id $ZENODO_ID --generate

# upload updated metadata (
python3 ./ --id $ZENODO_ID --upload

# deposit updated dummy file
python3 ./ --id $ZENODO_ID --deposit

# re-download and check what we think we deposited
python3 ./ --id $ZENODO_ID --retrieve
diff --report-identical-files

Send the Zenodo entry from the sandbox to for proofreading, include in "do you want to be included" email to contributors.

final Zenodo entry

This replicates the latter steps of the draft to create a new entry on the real Zenodo site. Do so *only* after the draft has been reviewed since it cannot be undone.


# note down ID of newest (topmost) "Einstein Toolkit"
ZENODO_ID=$(python3 ./ --list | perl -n -e '/(\d*) The Einstein Toolkit/ && print ($1) && exit 0')

# publish new version (cannot be undone)
python3 ./ --id $ZENODO_ID --publish

At this point you can download the bibtex entry from the Zenodo link. With a few minor edits it should be suitable for inserting into the einsteintoolkit.bib in the manifest.

update version information

  • update version information in flesh/Makefile in the CCTK_VERSION_XXX variables
  • update version information in flesh/doc/latex/cactus.sty in the \cactustitlepage command
export CCTK_VERSION=Cactus_4.X.0
CCTK_VERSION_MINOR=$(echo $CCTK_VERSION | cut -f2 -d.)
sed -i 's/\\cactustitlepage}\[4\]\[4\..*\]/\\cactustitlepage}[4][4.'$CCTK_VERSION_MINOR']/' doc/latex/cactus.sty
git add Makefile doc/latex/cactus.sty
git commit -m 'Cactus: update version in information in Makefile and docs'
  • update wvuthorns/Baikal/doc/documentation.tex and wvuthorns/BaikalETK/doc/documentation.tex with git hash of nrpytutorial used to generate code. The hash is mentioned in the README file (see below for instructions).

regenerate files

  • regenerate McLachlan, WeylScal4, CTThorns, and EinsteinExact from their source with the current version of Kranc by running make -j NJOBS in their respective m directory.
  • check that BaikalVaccum and Baikal can be regenerated using the nrpytutorial hash listed in their README file using cd BaikalETK && make
  • Next run the following script, which
    • generates the Cactus configure script by running autoconf2.13 in lib/make
    • generates the Cactus loop macros by running perl in src/include
    • generates UsersGuide, ReferenceManual and MaintGuide pdf files by running make UsersGuide ReferenceManual MaintGuide
cd lib/make
# commit changed files
git commit -a -m 'Cactus: regenerate configure' || true
cd ../../src/include
# commit changed files
git commit -a -m 'Cactus: regenerate cctk_Loop.h' || true
cd ../../
make UsersGuide ReferenceManual MaintGuide
cd doc
# commit changed files
git commit -a -m 'Cactus: regenerate documentation' || true

create branches

  • Important! Before this step, add the new bibtex entry to einsteintoolkit.bib in
  • Create release branches for all repositories in the ET
    • A new ET branch ET_YYYY_MM
    • A new ET tag ET_YYYY_MM_v0
    • A new Cactus branch Cactus_4.X.0 if in CactusCode
    • A new Cactus tag Cactus_4.X.0_v0 if in CactusCode
  • Note: SelfForce-1D is not automatically downloaded with GetComponents, so the repo can either be manually cloned into the repos directory or the branch and tag can be made and pushed separately from the script
# This script is run from inside the repos directory
export CCTK_VERSION=Cactus_4.X.0
set -x -e
for repo in */.git ; do
  cd $repo/..
  if git branch -a | grep -q origin/$ET_RELEASE ; then
    echo "Already pushed"
    cd - >/dev/null
  if ! grep -q 'fetch  *= *[+]refs/heads/[*]' .git/config ; then
    # unshallow in case --no-shallow was not passed to GetComponents
    git fetch --unshallow
    git config remote.origin.fetch "+refs/heads/*:refs/remotes/origin/*"
    git fetch origin
  repo_name=$(basename $PWD)
  git checkout -b $ET_RELEASE
  if [ $repo_name = manifest ] ; then
    sed -i '/!URL/a \!REPO_BRANCH = $ET_RELEASE'
    sed -i 's!/trunk!/branches/$ET_RELEASE!'
    git add
    git commit -m ' use release branch'
  git tag ${ET_RELEASE}_v0
  git push --tags --set-upstream origin $ET_RELEASE
  if git remote get-url origin | grep cactuscode ; then
    git branch $CCTK_VERSION
    git tag ${CCTK_VERSION}_v0
    git push --tags --set-upstream origin $CCTK_VERSION
  cd - >/dev/null
  sleep 5 # bitbucket enforces some rate limit on pushes its seems

update websites and online documentation

  • Update main ET web site (update version numbers, urls), searching for ET_YYYY_MM finds them all

set -e -x

export EDITOR=vim

export ET_RELEASE=ET_2020_05


export ET_RELEASE_URL=""

export ET_RELEASE_DATE="May 31st, 2020"
export ET_RELEASE_DATE_ISO=2020-05-31

ALL_FILES=$(git grep -l "\($PREVIOUS_ET_RELEASE\|$PREVIOUS_ET_RELEASE_CODENAME\)"  | grep -v 'about/releases/' | grep -vF past-releases.html | grep -vF publications/2013_MHD/index.php | grep -vF developers. | grep -vF members.txt | grep -vF mp.html) 

# update URL first since it tends to include the release codename
sed -i "s!$PREVIOUS_ET_RELEASE_URL!$ET_RELEASE_URL!g" index.html

# update release names

# release date
sed -i "s/[(]released on [^)]*[)]/(released on $ET_RELEASE_DATE)/g" download.html

# update past reelases, mark this one as most recent
# TODO: use something other than sed for this
sed -i -e '/List of Einstein Toolkit releases:/{n;b add_release}' -e 'b end' -e ':add_release;a\
          <li>'$ET_RELEASE' "'"$ET_RELEASE_CODENAME_FULL"'", <a href="about/releases/'$ET_RELEASE'_announcement.html">released '$ET_RELEASE_DATE_ISO'</a>\
          </li>' -e ':end' past-releases.html

# add news item to front page
$EDITOR index.html

git diff

git add $ALL_FILES past-releases.html download.html index.html
git commit -m "update to $ET_RELEASE"

echo 'Use git show and git commit --amend to correct mistakes'
:* Update
sed -i "s/$PREVIOUS_ET_RELEASE/$ET_RELEASE/g" simfactory/download/index.php
sed -i "s/$PREVIOUS_ET_RELEASE/$ET_RELEASE/g" CactusTutorial.ipynb
sed -i "s/kuibit==[0-9.]*/kuibit==$KUIBIT_RELEASE/" CactusTutorial.ipynb

From the repo, do the following

cd tutorial-server
sed -i "s/ENV ET_RELEASE.*/ENV ET_RELEASE ${ET_RELEASE}/" base.docker
git add base.docker
git commit -m "All changes for the $ET_RELEASE release"
touch variables.env
docker-compose -f docker-compose.base.yml build --no-cache --pull 2>&1 | tee docker-compose.base.log
docker-compose -f docker-compose.base.yml push
for c in notebook cilogon cyol ; do
    docker-compose -f docker-compose.$c.yml build 2>&1 | tee docker-compose.$c.log
    docker-compose -f docker-compose.$c.yml push

The tutorial server at checks for a new image once a day.

  • To update git clone

set -e -x






ET_DATE="$(date +"%B %d, %Y")"
AUTHOR="$(git config --get"

cd media/news

# keep the mkdir below to so that it fails if the output directories already
# exist

mkdir -p $ET_RELEASE
sed -i "s!^### .*!### $ET_DATE $AUTHOR!" $ET_RELEASE/

mkdir -p cactus_$CACTUS_VERSION
sed -i "s!^### .*!### $ET_DATE $AUTHOR!" cactus_$CACTUS_VERSION/

cat >../../_data/news.yml <<EOF
- date: "$ET_DATE"
  link: "$ET_RELEASE/index.html"
  title: 'Einstein Toolkit "$ET_NAME" Release'
- date: "$ET_DATE"
  link: "cactus_$CACTUS_VERSION/index.html"
  title: 'Cactus $CACTUS_VERSION Release'

git add ../../_data/news.yml $ET_RELEASE/ cactus_$CACTUS_VERSION/
git commit -m "news: add $ET_RELEASE announcement"

release anouncement issues

Other stuff

Changes to announce

First grab the latest dev version:

 curl -kLO
 chmod a+x GetComponents
 ./GetComponents --no-shallow --parallel

Then in that directory, the following script may be helpful in figuring out what has changed since the last release:

FORMAT="%h %ae %an: %s"
echo -n > .people.txt
for i in repos/*/.git
  export GIT_DIR=$i
  DEFAULT_BRANCH=$(git symbolic-ref refs/remotes/origin/HEAD | sed 's@^refs/remotes/origin/@@')
  echo "========================="
  echo "CHANGES FOR $(dirname $i)"
  git fetch --tags 1>/dev/null 2>/dev/null
  git log --format="$FORMAT" ${TAG}..${DEFAULT_BRANCH} -- 2>/dev/null
  git log --format="%an %ae" ${TAG}..${DEFAULT_BRANCH} 2>/dev/null >> .people.txt
  if [ $E != 0 ]
    # Pick your desired behavior in the absense of tags
    echo " >> TAG ${TAG} missing for repo, using date."
    git log --format="$FORMAT" $(git rev-list -n 1 --before="${DATE} 00:00" ${DEFAULT_BRANCH})..${DEFAULT_BRANCH} --
    git log --format="%an %ae" $(git rev-list -n 1 --before="${DATE} 00:00" ${DEFAULT_BRANCH})..${DEFAULT_BRANCH} -- >> .people.txt 2>/dev/null
    #echo " >> TAG ${TAG} missing for repo, assume it's a new contribution"
    #git log --format="$FORMAT" --
    #git log --format="%an %ae" -- >> .people.txt 2>/dev/null
done |& tee full.txt
sort -u < .people.txt > people.txt
rm .people.txt
cat people.txt

Note that when analyzing the output from the *nrpytutorial/* repo, you'll need to be careful to only check for changes to NRPyPN; all other changes are not ET related.

This misses changes in subversion repositories, currently only in ExternalLibraries which can be queried manually using svn log and usually contain very few changes.

This misses new repositories (or at best only considers recent changes to them).

Emails to send to contributors and gallery runners



In preparation for the upcoming "$RELEASE" release of the Einstein Toolkit I am collecting information about all who contributed to this release of the Einstein Toolkit. All contributors will be listed as authors as shown in this draft$ZENODO

You are shown as a contributor to $CODE in


Since you therefore contributed to this release, would you agree to be listed as an author in this and all future releases of the Einstein Toolkit?

If you agree to be listed I will add the information found here

to Zenodo.

Please let me or any of the other maintainers know, no later than $DEADLINE_DATE, if you agree to be included as an author and whether the information shown is correct. If there are additional persons that should be listed as authors, please let myself or one of the other maintainers know soon, so that they can be contacted.


To the main contributor (champion) send a request for citations


For each component the Einstein Toolkit keeps track of requested ("required") and suggested ("optional") citations that those who use the component should cite.

These are listed on:

and auto-generated from the main bibtex file (einsteintoolkit.bib) using the suggested-for and requested-for tags.

For you contributions, if you have any, please send me bibtex entries (in SPIRES style, see the beginning of

for the desired format) of any requested and suggested citations.

Typically we would suggest one requested citation and any number of suggested citations (eg for optional features or secondary citations).

Websites and Zenodo DOIs are fine, it does not have to be a published paper (though having a DOI would be good).


Release team


in preparation for the upcoming "$RELEASE" release of the Einstein Toolkit I am collecting information about the release team for this release of the Einstein Toolkit. All release team members will be listed as authors in$ZENODO

Since you are a member of the release team, would you agree to be listed as an author in this release of the Einstein Toolkit?

If you agree to be listed I will add the information found here

to Zenodo.

Please let me or any of the other maintainers know, no later than $DEADLINE_DATE, if you agree to be included as an author and whether the information shown is correct. If there are additional persons that should be listed as authors, please let myself or one of the other maintainers know soon, so that they can be contacted.