Services

The Einstein Toolkit depends on a number of services managed by the ET maintainers. Here we list each of these, and give any relevant information. Ideally, this page should be sufficient for a maintainer to deal with urgent issues relating to these services in case the primary administrator is unavailable.

On 2018-12-10 and 2018-12-17 Steve Brandt and Roland Haas gave a short introduction to the ET infrastructure services. Here are the slides used which may contain information not yet included in this wiki.

Getting Help

• For LSU sites, you can email CCT IT Support <it-support@cct.lsu.edu>, please mention that you work with Steve Brandt. Ideally, you should have a CCT external user account when sending help mail. If you don't have one, please email the above it-support@cct.lsu.edu address to ask for one.

• For NCSA, the generic email for help of any kind is help@ncsa.illinois.edu and CC'ing Roland Haas (rhaas@illinois.edu) will help. If you are involved in a service hosted at NCSA we can create an external collaborators NCSA account for you, please contact Roland to create such an account.

Website

• https://einsteintoolkit.org
• Location: A VM at CCT
• Administrator: Steve Brandt
• Source for website: https://bitbucket.org/einsteintoolkit/www.git
• Docker source for webserver: https://github.com/stevenrbrandt/et-websites.git
• Updating the website:
  • Login to the einsteintoolkit.org server as root
  • Go to the et-websites directory
  • Type 'docker-compose build'
  • Type 'docker-compose down' to bring down the server
  • Type 'docker-compose up -d' to restart with the new image
  • the website docker container auto-updates once a day
• Members list: The members list is updated manually, in response to an email sent by the website's PHP script. The email sends the subscribers email address, whether they'd like to be added to the mailing list, a link to add the user to the mailing list, and a text entry for members.txt (from which the members page is generated). In addition, there is a python script, sortmembers.py, which sorts the members.txt file.
• Release announcements: Each must be put in the about/releases directory, using the name ET_{year}_{month}_announcement.md. The script mk_release_announcements.py will generate a txt and html file from the md file.
• the website reports unanswered question candidates on https://www.einsteintoolkit.org/tools/unanswered.php which is driven by the script unanswered.pl in the tools directory of the www repo. Occasionally the script may be unable to contact the wiki page https://docs.einsteintoolkit.org/et-docs/Answered_emails that holds emails flagged as "answered". When this happens only from within the webserver then Roland and Steve observed (at least once) that the ARP cache on the webserver was incorrect and could be reset by contacting the webserver from the wiki server

ThornGuide on einsteintoolkit.org

• ThornGuide.php is in /www/einstein/www/documentation
• It draws on the autogenerated http doc files, which are updated once a day by a GitHub action docs.yml
• the github action trigger the webserver to update itself

Tutorial

• Source: https://github.com/nds-org/jupyter-et
• Found at: https://etk.cct.lsu.edu/
• Access to server: Steve Brandt
• Accounts creation: Steve Brandt, Roland Haas via ET website repo

git clone  git@bitbucket.org:einsteintoolkit/www
cd www
python3 ./whitelist-code-gen.py  email@example.com
# this adds a md5 hash of the email address to the file tutorials-whitelist.txt
git add tutorials-whitelist.txt 
git commit -m 'Tutorial whitelist: new user added'
git push
# respond to request on mailing list *and* to requestor directly

Wiki

• https://docs.einsteintoolkit.org/et-docs/
• Location: A VM and Database at CCT
• Accounts:
  • Log into docker image as root (docker exec -it wiki bash)
  • To create a user named dvader with password dieoobiwandie, run the command php /var/www/mediawiki/maintenance/createAndPromote.php dvader dieobiwandie
  • To change a password, run the command php /var/www/mediawiki/maintenance/changePassword.php --user=dvader --password=iluvupadme
• Administrator: Steve Brandt, Roland Haas
• Source code: https://github.com/stevenrbrandt/et-websites.git

git

• EinsteinToolkit bitbucket organization: https://www.bitbucket.org/einsteintoolkit, list of admins
• EinsteinToolkit github organization: https://github.com/einsteintoolkit, list of admins

The organizations should has administrator group and a developer group where administrators are the ET maintainers and developers are ET users who have been granted write access to the repos. We try to keep access control coarse grained, erring on the side of granting write access easily and to all repos rather than micromanaging access.

New code repositories on bitbucket should set up a "repo push" (the default) web-hook to contact https://einsteintoolkit.org/hooks/repo_changed.php?secret=XXX to send commit messages to commits@einsteintoolkit.org where XXX is to be found in https://bitbucket.org/einsteintoolkit/www/src/master/hooks/repo_changed.php#lines-18 (or thereabouts, look for the variable $secret).

New code repositories on github should set up commits@einsteintoolkit.org as the recipient of push event emails in https://github.com/GITHUB_USER_NAME/REPO_NAME/settings/notifications/edit .

github is used by gitter.im to authenticate users but access to https://gitter.im/einsteintoolkit is possible without being in the github einsteintoolkit organization.

Bitbucket ticket repository

• https://trac.einsteintoolkit.org
• a regular bitbucket repository at https://bitbucket.org/einsteintoolkit/tickets owned by the einsteintoolkit organization
• default page has been changed to be "Issues" rather than code
• a web-hook has been set up to contact https://einsteintoolkit.org/hooks/issued_changed.php?secret=XXX for ticket change notifications where XXX is to be found in https://bitbucket.org/einsteintoolkit/www/src/master/hooks/issue_changed.php#lines-50
• conversion from trac used a File:tickets mod.python of the trac2bitbucket code.

Mailing lists

• http://lists.einsteintoolkit.org/mailman/listinfo
• Location: A VM at CCT, hostname lists.einsteintoolkit.org port 2525 reachable via bounce.cct.lsu.edu
• Administrator: Steve Brandt. He needs to filter the spam daily. For those who want to know, Russian brides are still a popular subject in spam.
• archives wrap text emails at the 80 character mark using CSS in /etc/mailman/templates/en/article.html

   <STYLE TYPE="text/css">
     pre {
           white-space: pre-wrap;       /* css-2.1, curent FF, Opera, Safari */
           white-space: -moz-pre-wrap;
           white-space: -pre-wrap;
           white-space: -o-pre-wrap;
           word-wrap: break-word;
           width: 80ch;
           overflow-x: auto;
     }
   </STYLE>

• ensuring that mailman's aliases are seen by postfix requires following instructions on https://www.gnu.org/software/mailman/mailman-install/postfix-integration.html#SECTION001611000000000000000 and setting alias_map in main.cf to::

alias_maps = hash:/etc/aliases, hash:/etc/mailman/aliases

Have to force-allow some SMTP servers (Google.com in particular) for postdfix, currently in a file /etc/postfix/google_acces and /etc/postfix/main.cf

check_client_access hash:/etc/postfix/google_access,
check_sender_access hash:/etc/postfix/google_access,

which must appear in *both* smtpd_relay_restrictions and smtpd_recipient_restrictions settings.

The access file google_access reads

google.com OK
amantadine.ncsa.uiuc.edu OK
outbound.protection.outlook.com OK
id.atlassian.com OK
mail-us.atlassian.net OK

After modifying which one must run:

sudo/usr/sbin/postmap google_access
sudo service postfix restart

Logs of postfix appear in journalctl:

journalctl -f -t postfix/smtpd

Chat channels

• IRC channel #cactus on irc.oftc.net
• Gitter channel https://gitter.im/EinsteinToolkit access is controlled using https://github.com accounts

Zoom

• we are using a zoom instance provided by LSU, details described on ET_call_Zoom_Info.
• Administrator: Steve Brandt

ET call reminders

• reminders to add to the agenda and for the calls themselves are sent to users@einsteintoolkit.org at Monday 15:18 Central time and Wednesday 17:15 Central time respectively, via cron jobs running on Roland Haas' workstation
• Administrator: Roland Haas

#!/bin/bash

if [ $(date +%s) -lt  $(date +%s -d  2019-06-21) ] ; then
exit
fi

mail -r users@einsteintoolkit.org -s "Agenda for Thursday's Meeting" users@einsteintoolkit.org <<EOF
Please update the Wiki with agenda items for Thursday's meeting. Thanks!

https://docs.einsteintoolkit.org/et-docs/meeting_agenda

--The Maintainers
EOF

#!/bin/bash

if [ $(date +%s) -lt  $(date +%s -d  2019-06-21) ] ; then
exit
fi

mail -r users@einsteintoolkit.org -s "Einstein Toolkit Meeting Reminder" users@einsteintoolkit.org <<EOF
Hello,

Please consider joining the weekly Einstein Toolkit phone call at
9:00 am US central time on Thursdays. For details on how to connect
and what agenda items are to be discussed, use the link below.

https://docs.einsteintoolkit.org/et-docs/Main_Page#Weekly_Users_Call

--The Maintainers
EOF

CactusCode.org

• https://www.cactuscode.org
• Location: a GitHub pages setup https://github.com/einsteintoolkit/www.cactuscode.org
• DNS for www.cactuscode.org maintained by CCT/LSU

Website monitoring

Websites are monitored every 15 minutes by https://einsteintoolkit.github.io/uptime using a dummy github einsteintoolkit-uptime with email address maintainers+uptime@einsteintoolkit.org. Currently monitored pages are:

• https://www.cactuscode.org
• https://www.einsteintoolkit.org
• https://docs.einsteintoolkit.org

Email updates are sent to <maintainers@einsteintoolkit.org> which has had uptime@einsteintoolkit.org added to its list of allowed posters. An older hand-written monitor of Frank Loeffler's is at https://www.cct.lsu.edu/~knarf/cgi-bin/monitor.cgi .

Access to the monitors is by all ET maintainers via its github repo https://github.com/einsteintoolkit/uptime

Domain

• einsteintoolkit.org
• Owned by LSU
• Notes: Cannot point subdomains to any machines not controlled directly by LSU

Build and test

• github action at https://github.com/EinsteinToolkit/tests/blob/master/.github/workflows/main.yml
• scripts (currently) in gh-pages branch of tests repo
• publishes result to gh-pages: https://einsteintoolkit.github.io/tests
• workflow at https://github.com/EinsteinToolkit/tests/blob/master/.github/workflows/docs.yml runs once a day and updates thornguide etc on https://www.einsteintoolkit.org if there were changes to the flesh repo
  • triggers website update pull using https://www.einsteintoolkit.org/update.php and HTML tag in the repo to hold HTML tar archive

Adding a new thorn

The testing system https://github.com/EinsteinToolkit/tests/blob/master/.github/workflows/main.yml does not use GetComponents to check out the code. Instead it uses a git super-repository where all thorns are submodules.

One creates a new submodule in repos</repos> for each git repository, similarly to how GetComponents creates a git checkout for each repository. One then has to manually create symbolic links from <codde>arrangements/&langle;ARRANGEMENT&rangle;/&langle;THORN&rangle; to the checkout. The submodule must be set up to track the main development branch (main, master, etc.).

For example for CarpetX

git clone https://github.com/einsteintoolkit/tests
cd tests
git submodule update --recursive --init
cd repos
git submodule add -b main https://github.com/eschnett/CarpetX CarpetX

and then update .gitmodules to read

[submodule "repos/CarpetX"]                                                                                     path = repos/CarpetX
  revision = main
  branch = main
  url = https://github.com/eschnett/CarpetX
  upstreamurl = git@github.com:eschnett/CarpetX
  upstreamtype = git
</code>

And finally create the arrangements etc. use GetComponents

<pre>
cd tests
bin/GetComponents --no-update --root $PWD manifest/einsteintoolkit.th

then use git status to then git add and git commit</commit> the changes.

Git updater

• A pipeline running 15minutes wise on GitHub: https://github.com/EinsteinToolkit/tests/blob/master/.github/workflows/updater.yml
• uses a bot-user einsteintoolkit-updater with email maintainers+updater@einsteintoolkit.org
  • password held by Roland, but can be reset using the maintainers@einsteintoolkit.org mailing list (pw reset)
• its push triggers a GitHub action to run the tests: https://github.com/EinsteinToolkit/tests/blob/master/.github/workflows/main.yml