ET Workshop Summer 2013

From Einstein Toolkit Documentation
Jump to: navigation, search

New User's workshop

Summer Einstein Toolkit Workshop 2013

The users workshop is part of the CGWAS and will be held on the last day of the summer school (Friday). Currently the programme lists introduction to numerical relativity, an introduction to the ET and two hands-on sessions. The big hands-on session from last year was too technical and should be reworked if used at all. Instead we could consider running a very short head-on nsns run and analysing the data. The runs takes ~20min at a ridiculously low resolution, but might be sufficient to shows how it's done.

Spacetime Coordinates and Contact Information

The new users workshop is part of CGWAS and is held on the last day of the school (Friday).

Workshop email list:, emergencies: Frank Löffler (, +1-225-202-1752) or Roland Haas (, +1-404-386-3657)

List of Lecturers

Below are the ET volunteers who will be presenting or heading parts of the sessions.

Name Affiliation
Roland Haas Caltech Introduction to the ET
Erik Schnetter Perimeter Institute Introduction to numerical relativity
Frank Löffler LSU Introduction to the ET
Joshua Faber RIT hands-on session
Steve Brandt LSU hands-on session


Day Time Activity Presenter / Responsible
Fri 9:00-10:30 Introduction to Numerical Relativity Erik Schnetter
Fri 10:30-11:00 Break
Fri 11:00-11:40 Introduction to the Einstein Toolkit, Overview Frank Löffler
Fri 11:40-12:30 Introduction to the Einstein Toolkit, Details Roland Haas
Fri 12:30-14:00 Lunch
Fri 14:00-14:20 Introduction to hands-on environment: simfactory and installing VisIt Steven Brandt
Fri 14:20-14:40 Introduction to hands-on Cactus thorn-writing Roland Haas
Fri 14:40-15:00 Introduction to hands-on visualization within the Einstein Toolkit Roland Haas
Fri 15:00-15:30 Break
Fri 15:30-17:30 Hands-on, center of mass for analysis purposes (simple way) everyone helping
Fri 17:30-18:00 Examples of ET production usage Christian Reisswig


Hands on exercises

Running and extending (par-file) of two provided examples: perturbed, single BH, and perturbed, single TOV star
Hands-on: Using visualization tools to look at meaningful quantities ( TOV profile evolutions ) (slides)
Create movie of either BH+GW or TOV oscillations using VisIt (slides, TOV session file, TOV data file, Psi4 session file, Psi4 data file)
Hands-on: Implement your own thorn (neutron star tracking), and visualize it, in teams of about 5 people (, more information at end of wiki

Developer's workshop

ET Workshop Summer 2013 (Developers Workshop

Developers' workshop

Spacetime Coordinates and Contact Information

Dates Sat July 27th - Mon 29th (incl.)
Location Cahill room 370, H323 call ins from AEI, TAT. (call in instructions)
Contacts Workshop email list:, emergencies: Frank Löffler (, +1-225-202-1752) or Roland Haas (, +1-404-386-3657)
Hotel Saga Motor Inn, 1633 E. Colorado Blvd., Pasadena, CA 91106, Preferred reservation line, Phone (626) 795-0431, Fax (626) 792-0559,, $65.00 double bed, one person, $70.00 double bed, two people, Caltech rate (mention when calling)

Vagabond Inn, Pasadena, 1203 East Colorado Blvd., Pasadena, CA 91106, Phone (626) 449-3170,, $79.00 double bed (one or two persons), rate on website differs, best to call and mention Caltech.

Transportation - Hotel There are many possible ways of getting from the airport to Pasadena:
  • Taxi (~$100)
  • Super Shuttle (~$40)
  • FlyAway Bus + Metro Gold Line (~$9 in total)

At any terminal at LAX, you will find departure platforms for all of the above transportation systems just outside the terminals.

The FlyAway Bus + Metro is usually very convenient. To get on it, just wait by the blue sign that says "FlyAway Busses". Wait until a FlyAway Bus with sign "Union Station" arrives (it departs every ~30min). Jump on it and go until Union Station (its only and final destination). You pay $7 upon arrival (they have change if you don't exactly have $7). At Union Station, go to the platform that says "Metro Gold Line". Pay $1.50 at the ticket machine (one way ticket). Take the Metro Gold Line (direction "Sierra Madre Station") and exit at "Lake Station" (Caltech or Vagabond Inn) or "Allen Station" (best for the Saga Motor Hotel). Check out google maps for further info. Here's a link to Metro Gold Line, including a link to this trip planer (it also knows about FlyAway busses).

Transportation - Workshop walking distance, directions.
Food options Chandler's on campus, Ernie's food truck (times), restaurants on Lake, anything on Colorado.
h323 Instructions

The room's IP address is, it should accept calls. PLEASE DO NOT TEST this right now since the room is often in use for group meetings etc. Instead check the room booking page and send an email to Roland Haas who will operate to room for a test.

We have had best success so far by connecting from another conference room system (Tandberg, Polycom seem to work). Failing that it is possible to use Ekiga for Linux (call h323: However we found that the versions of Ekiga in Ubuntu and Debian require specific settings to work. In particular the only audio/video codec combination that seems to work is G722 for audio and H261 for video. Ekiga lets you enable/disable codecs in its options dialog. We do not know of a working client for OSX or Windows at this point (XMeeting claims to be H323 client for OSX).


for the developers' workshop (July 27th to 29th):

Name Affiliation Hotel
Steven Brandt Louisiana State University Vagabond
Peter Diener Louisiana State University Vagabond
Josh Faber RIT Saga
Roland Haas Caltech local
Frank Löffler Louisiana State University Vagabond
Philipp Mösta Caltech local
Christian Ott Caltech local
Ian Hinder Albert Einstein Institute Remote
Tanja Bode Universität Tübingen Remote

In-Workshop Details

Want to have a relatively large list of possible topics then select which ones to actually work on.

Ideas for topics:

  • Vectorization
  • AMR grid setup
  • AMR scaling
  • IO scaling
  • Data-driven scheduler
  • Cleanup of GRHydro
  • Automated performance tests to verify that changes are actually beneficial
  • MHD development:
    • Refluxing
    • A-field MHD
  • David Rideout offered to give a presentation
  • McLachlan: Erik and Ian want to make several changes to the code:
    • Allow arguments to be passed via scripts to generate multiple thorns in parallel, rather than sequentially
    • Design of more high level APIs to streamline declaration of new objects, e.g. tensors
    • Introduce master calculation for BSSN variables, and use filtering to isolate necessary code to be called in a specific run
    • Enable making parameters available at either runtime or Kranc time for greater flexibility
    • Additional optimizations


Day Time Activity Presenter / Responsible


  • Wireless access is through the (dedicated) CGWAS network.



Todo for sure:

  • DR talk
  • Chemora

list of votes on topics:

  • (3) MHD, A, s
  • (1) GRHydro cleanup
  • (3) GRHydro vectorization
  • (2) McLachlan
  • (3) scaling
  • (2) benchmarks


  • Ian: Paper recently added to arxiv:
  • got funded (LSU, GaTech, NCSA, [Beijing])
    • Money should benefit the whole Chemora group. Possible get together in Atlanta as Zhihui Du, our Chinese collaborator, will be there for the next year or so. Other venues are possible.
  • "PDE" compiler
    • Kranc, maybe with Hydro -> CaKernel
    • Need more language features in the PDE language
  • Use of accelerators (and CPUs)
    • OpenCL (for all CUs) - Want this to make sure we can run on any new machine.
    • CUDA - Want to be able to use this. CaKernel can do CUDA or OpenCL.
  • Machine profile
    • hwloc
    • PAPI (now also works on Intel MICs) - dynamic feedback for LoopControl or other things.
  • Production use
  • Not just for exascale
    • The concern is that a wide variety of big machines be enabled, too many proposals were targeting exascale.
  • Intermediate Black Hole Binary Science Driver - Gatech primarily responsible. However, the Chinese side recently put in a proposal to Titan for this. Getting benefits in GRHydro may, however, be more important.
  • Load Balancing - We'll need Peter's load balancer. Using AMPI with Cactus was also promised.
  • First steps
    • Get the code working again
    • Run on one node
    • Focus on Supermike 2
  • Mailing list (hackathon)
  • Website


  • Quote of the day: Christian Ott: "Have you solved the scaling problem yet?"
  • Carpet switched to new internal representation of the box-datastructure, improving scaling of modify-operations of this (initial setup, regridding)
  • Since Hydro isn't vectorized, we loose performance on vector machines, unsurprisingly.

David Rideout's Talk

  • Directions for Cactus HPC Framework
  • Tension between generic nature of framework vs. scientific results
  • Tangent discussion: need for elliptic solver to increase userbase, but hard
    • More on the tangent: whether to use packages or write from scratch
    • Ian: See which will be free software soon. This is a generic multigrid elliptic solver using Carpet.
  • Big Data: esp. phone data is important current research topic
  • Cosmological simulations can be modeled using similar techniques as networks
  • Relevance to Cactus
    • Network data as discrete spacetime model
    • Discrete quantum gravity
    • Unstructured Meshes
  • Changes to Cactus to incorporate different grid structures and models may be quite significant, limited experience for current maintainers

Erik Schnetter's Paper

  • Performance and Optimization Abstractions for Large Scale Heterogeneous Systems in the Cactus/Chemora Framework: Media:xscale2013.pdf

Staggered A-field Evolution

In the staggered system, B-fields live on faces (gets its own gridfunction) A-field lives on edges (centered versions get a new variable) E-field lives on edges, but may be stored locally (?) reconstruction needs to propagate some quantities, like velocity from centers to edges.

  • Changes were made to interfaces and params, and committed to Zelmani (may change as needed)
  • Tasks to work on
    • Prolongation/restriction/carpetLib operators -- Philipp
    • HLLE for the Avec, being careful to consider which variables live where -- Philipp
    • Reconstruction for Bvec, velocities (doubly so) at edges -- Frank/Tanja
    • Calc Update, Source Terms, averaged B field for fluid fluxes -- Josh
    • Documentation of the method -- Josh/???