Release Process

From Einstein Toolkit Documentation
Revision as of 21:29, 29 September 2020 by Sbrandt (talk | contribs) (→‎Other stuff)
Jump to: navigation, search

Release Process

This page describes the release process in some detail. Getting a release out of the door without embarrassing oversights requires a bit more planning and care than one would like.

TODO: reformat this page as a checklist which can be marked up for each release as each item is addressed

Long-term planning

A few months or weeks before the release:

  • Decide it is time to make another release
  • form a release coordination team
    • a release coordinator
    • a release coordinator assistant, who may be the release coordinator for the next release
    • the former release coordinator to help out
  • Choose features which are going to be included, and those which won't be included
  • Choose a tentative date
  • Begin discussions on the mailing list, reminding people to look at test cases, review patches etc.

Two months before the release

  • Set up a wiki planning page for the release
  • Review thorns for code quality, documentation, test cases
    • Ask for review volunteers, assign tasks to people
    • Remove new features without reviewer from list of new features
  • Decide on a thorn list
  • Choose a concrete date

One months before the release

  • Ask for example runner volunteers, assign tasks to people
  • Update einsteinttoolkit.bib with requested and suggested citations for all thorns in thorn list
  • Decide on a list of release-critical systems
  • Collect list of new features, newsworthy items, and acknowledgements for release announcement on wiki
  • Announce date publicly

Two weeks before the release

  • feature freeze, no more new features accepted
  • verify that gallery runs and example parfiles still work with new release
  • contact contributors and example runners about permanent (contributors) and one-off (example runners) inclusion in the ET author list
  • announce final member list of release team:
    • release coordinator(s)
    • test runner(s)
    • gallery example runner(s)
    • reviewer(s)
    • anyone putting in work specifically for the release
  • Draft release announcement
  • set up draft Zenodo entry:
# get Steve Brandt's personal access token for zenodo.org
# create your own personal access token for sandbox.zenodo.org: https://sandbox.zenodo.org/account/settings/applications/tokens/new/

# get data out of current Zenodo entry
export ZENODO_ACCESS=$(secret-tool lookup hostname https://zenodo.org username sbrandt@cct.lsu.edu) python3
python3 ./zenodo.py --list
# note down ID of "Einstein Toolkit" (will change each release)
python3 ./zenodo.py --id 3522086
mv zupload.py et-zupload.py
wget "https://zenodo.org/record/3522086/files/Definition.md?download=1" -O Definition.md

# recreate in sandbox
export ZENODO_ACCESS=$(secret-tool lookup hostname https://sandbox.zenodo.org username rhaas@illinois.edu)
python3 ./zenodo.py --sandbox --create et-zupload.py
# note down ID of newest (topmost) "Einstein Toolkit"
python3 ./zenodo.py --sandbox --list
python3 ./zenodo.py --sandbox --id 587417 --deposit Definition.md
python3 ./zenodo.py --sandbox --id 587417 --publish

# update metadata and dummy deposit file (file must change or Zenodo will reject the new version)
sed -i 's/ET_2019_10/ET_2020_05/;s/c32f345352864d88cb4fa6e649262d35da69a1a7/ET_2020_05_v0/g'  Definition.md 
python3 ./zenodo.py --sandbox --id 587417 --newversion
python3 ./zenodo.py --sandbox --list
python3 ./zenodo.py --sandbox --id 587432 --deposit Definition.md
python3 ./zenodo.py --sandbox --id 587432 --upload
python3 ./zenodo.py --sandbox --id 587432 --publish

One or two days before the release

  • Have a telecon with the release team, discuss (and modify if necessary) release process and responsibilities
  • Get a list of all repositories that are involved (including repositories for tools such as GetComponents)
  • Publicly declare a freeze on all involved thorns and tools
  • Ensure that any manually-generated files are consistent
    • regenerate McLachlan, WeylScal4, CTThorns, and EinsteinExact from their source with the current version of Kranc
    • generate the Cactus configure script
    • generate the Cactus loop macros etc
    • generate UsersGuide, ReferenceManual and MaintGuide pdf files
  • Test all thorns on all systems, collect status reports on wiki
  • verify that new users tutorial still works
  • Update release planning wiki page with peoples', thorns', and systems' statuses
  • Set up a (smaller) technical release team who will be available all day on the day of the release

The release

  • Have the technical release team meet in the same room (brick, EVO, chat, phone)
  • Briefly re-check status of thorns and machines
  • Possibly disable all outside write access to all critical repositories
  • update version information:
    • update version information in flesh/Makefile in the CCTK_VERSION_XXX variables
    • update version information in flesh/doc/latex.sty in the \cactustitlepage command
    • update wvuthorns/Baikal/doc/documentation.tex and wvuthorns/BaikalETK/doc/documentation.tex with git hash of nrpytutorial used to generate code. The hash is mentioned in the README file. To compare codes, check out https://github.com/zachetienne/nrpytutorial.git at the hash in question and generate using ./run_Jupyter_notebook.sh Tutorial-BaikalETK.ipynb. In ET_2020_05 this required forcing python3. The generated code lacks using CCTK_REAL for non-vectorized code, and the order in which finite difference orders are checked and included is timing dependent and thus random.
  • Update component list einsteintoolkit.th in *master* before tagging it for release branch to point to new stable version
sed -i "/!DEFINE  *COMPONENTLIST_TARGET/a \\\\n!DEFINE ET_RELEASE = $ET_RELEASE" einsteintoolkit.th
sed -i '/!URL/a \!REPO_BRANCH = ET_RELEASE' einsteintoolkit.th
sed -i 's!/trunk!/branches/$ET_RELEASE!' einsteintoolkit.th
  • undo changes on master branch
  • Create release branches for all repositories in the ET
    • A new ET branch ET_YYYY_MM
    • A new ET tag ET_YYYY_MM_v0
    • A new Cactus branch Cactus_4.X.0 if in CactusCode
    • A new Cactus tag Cactus_4.X.0_v0 if in CactusCode
#!/bin/bash
export ET_RELEASE=ET_XXXX_YY
export CCTK_VERSION=Cactus_4.X.0
set -x -e
for repo in */.git ; do
  cd $repo/..
  pwd
  git checkout -b $ET_RELEASE
  git tag ${ET_RELEASE}_v0
  git push --tags --set-upstream origin $ET_RELEASE
  if git remote get-url origin | grep cactuscode ; then
    git branch $CCTK_VERSION
    git tag ${CCTK_VERSION}_v0
    git push --tags --set-upstream origin $CCTK_VERSION
  fi
  cd - >/dev/null
done

export ET_RELEASE=ET_XXXX_YY
git checkout -b $ET_RELEASE
sed -i "s/master/$ET_RELEASE/" .gitmodules
git add .gitmodules
git submodule foreach bash -c "git fetch ; git checkout origin/$ET_RELEASE"
git commit -am "Create release branch for $ET_RELEASE"

#!/bin/bash

set -e -x

export ET_RELEASE=ET_2020_05
export PREVIOUS_ET_RELEASE=ET_2019_10

export ET_RELEASE_CODENAME=Turing
export PREVIOUS_ET_RELEASE_CODENAME=Mayer

export ET_RELEASE_URL="https://en.wikipedia.org/wiki/Alan_Turing"
export PREVIOUS_ET_RELEASE_URL="https://en.wikipedia.org/wiki/Maria_Goeppert_Mayer"

export ET_RELEASE_DATE="May 31st, 2020"

ALL_FILES=$(git grep -l "\($PREVIOUS_ET_RELEASE\|$PREVIOUS_ET_RELEASE_CODENAME\)"  | grep -v 'about/releases/' | grep -vF past-releases.html | grep -vF publications/2013_MHD/index.php | grep -vF mp.html) 

# update URL first since it tends to include the release codename
sed -i "s!$PREVIOUS_ET_RELEASE_URL!$ET_RELEASE_URL!g" index.html

# update release names
sed -i "s/$PREVIOUS_ET_RELEASE/$ET_RELEASE/g" $ALL_FILES
sed -i "s/$PREVIOUS_ET_RELEASE_CODENAME/$ET_RELEASE_CODENAME/g" $ALL_FILES

# release date
sed -i "s/[(]released on [^)]*[)]/(released on $ET_RELEASE_DATE)/g" download.html

# update past reelases, mark this one as most recent
$EDITOR past-releases.html

# add news item to front page
$EDITOR index.html

git diff

git add $ALL_FILES past-releases.html download.html index.html
git commit -m "update to $ET_RELEASE"

sed -i "s/$PREVIOUS_ET_RELEASE/$ET_RELEASE/g" index.php

sed -i "s/$PREVIOUS_ET_RELEASE/$ET_RELEASE/g" CactusTutorial.ipynb
  • log into etkhub.ndslabs.org, pull the repo, and rebuild the docker image
  • once release documentation finishes building on Jenkins, trigger update of ET website
  • Check out release branch on all systems, re-run all quick tests
cd media/news

cp -a ET_2019_10 ET_2020_05
sed -i 's!Maria_Goeppert_Mayer!Alan_Turing!g' ET_2020_05/index.php
sed -i 's!Mayer!Turing!g' ET_2020_05/index.php
sed -i 's!>.*</h3>!>May 31, 2020 Roland Haas</h3>!' ET_2020_05/index.php

cp -a cactus_4.7.0 cactus_4.8.0
sed -i 's!4[.]7[.]0!4.8.0!g;' cactus_4.8.0/index.php
sed -i 's!Maria_Goeppert_Mayer!Alan_Turing!g' cactus_4.8.0/index.php
sed -i 's!Mayer!Turing!g' cactus_4.8.0/index.php
sed -i 's!ET_2019_10!ET_2020_05!g' cactus_4.8.0/index.php
sed -i 's!>.*</h3>!>May 31, 2020 Roland Haas</h3>!' cactus_4.8.0/index.php

mv recent.php tmp.php
head -n4 <tmp.php | \
sed 's!4[.]7[.]0!4.8.0!g;' | \
sed 's!Maria_Goeppert_Mayer!Alan_Turing!g' | \
sed 's!Mayer!Turing!g' | \
sed 's!ET_2019_10!ET_2020_05!g' | \
sed 's!Oct 2019!May 2020!' | \
cat tmp.php - >recent.php
rm tmp.php

After the release

  • Watch mailing lists for problem reports
  • Use released version to repeat a few production simulations
  • Update this page with new lessons learned

Helpful Tools

This section collects a handful of useful commands that may become critical to guarantee the quality of the release. The descriptions below are quite technical and intended for the developers and maintainers only.

Technical Details

On einsteintoolkit.org

  • ThornGuide.php is in /www/einstein/www/documentation
  • It draws on the autogenerated http doc files, which are found here:
 /var/www/einstein/documentation

 docker build --no-cache -f etk-website.docker -t stevenrbrandt/etk-website .
 docker push stevenrbrandt/etk-website


Other stuff

On https://stevenrbrandt@bitbucket.org/einsteintoolkit/www.git

  • Create new docs in about/releases/

On https://svn.cactuscode.org/www

  • Create the announcement in media/news/recent.php and associated links.

The following script may be helpful in figuring out what has changed since the last release: 

 MONTH=05
 YEAR=2020
 FORMAT=oneline
 ####
 TAG="ET_${YEAR}_${MONTH}_v0"
 DATE="${MONTH}/01/${YEAR}"
 for i in repos/*/.git
 do
   export GIT_DIR=$i
   echo "CHANGES FOR $(dirname $i)"
   git fetch --tags 1>/dev/null 2>/dev/null
   git log ${TAG}..master --format=$FORMAT 2>/dev/null
   E=$?
   if [ $E != 0 ]
   then
     echo " >> TAG ${TAG} missing for repo, using date."
     git log $(git rev-list -n 1 --before="${DATE} 00:00" master)..master --format=$FORMAT
   fi
 done

News announcement to users@einsteintoolkit.org, users@cactuscode.org, news@cactuscode.org, allcct@cct.lsu.edu

Post to Ad, news on http://hyperspace.uni-frankfurt.de/

Post announcement to the Einstein Toolkit Facebook page.

Update and commit the changes to the thornlist in the manifest 

   https://bitbucket.org/einsteintoolkit/manifest.git

Create the git tags and branches: ET_2018_02_v0, Cactus_4.4.0

  • in each git repo:
       git tag ET_2018_02_v0; git push origin ET_2018_02_v0
       git checkout -b ET_2018_02; git push origin ET_2018_02
  • use git fetch -t to get the latest tags.
  • Create svn tags and branches (no longer needed since ET_2019_03 "Proca")
 for i in BLAS LORENE GSL PETSc LAPACK FFTW3 pciutils  hwloc OpenBLAS MPI OpenCL HDF5 PAPI zlib libjpeg
 do
     cd /home/sbrandt/cactus/Cactus/arrangements/ExternalLibraries/$i
     svn copy https://svn.cactuscode.org/projects/ExternalLibraries/$i/trunk \
         https://svn.cactuscode.org/projects/ExternalLibraries/$i/tags/ET_2018_02_v0 \
         -m "Tagging release ET_2018_02"
done
for i in BLAS LORENE GSL PETSc LAPACK FFTW3 pciutils  hwloc OpenBLAS MPI OpenCL HDF5 PAPI zlib libjpeg
do
    cd /home/sbrandt/cactus/Cactus/arrangements/ExternalLibraries/$i
    svn copy https://svn.cactuscode.org/projects/ExternalLibraries/$i/trunk \
        https://svn.cactuscode.org/projects/ExternalLibraries/$i/branches/ET_2018_02 \
        -m "New branch for release ET_2018_02"
done


  • creating release branches
    • svn: svn copy https://<repository-server>/<repository-path>/trunk https://<repository-server>/<repository-path>/branches/ET_2011_05
    • darcs: clone the repository, appending a suffix to the name
    • git: git checkout -b ET_2011_05 ; git push --set-upstream origin ET_2011_05
Retrieved from "https://docs.einsteintoolkit.org/et-docs/index.php?title=Release_Process&oldid=7098"