Difference between revisions of "ET Workshop Summer 2013 (Developers Workshop)"
(→Spacetime Coordinates and Contact Information) |
(→Notes) |
||
| (42 intermediate revisions by 10 users not shown) | |||
| Line 1: | Line 1: | ||
| − | == Summer | + | == ET Workshop Summer 2013 (Developers Workshop == |
| − | === | + | === Developers' workshop === |
| − | |||
| − | |||
==== Spacetime Coordinates and Contact Information ==== | ==== Spacetime Coordinates and Contact Information ==== | ||
| Line 10: | Line 8: | ||
|- | |- | ||
! Dates | ! Dates | ||
| − | | ''' | + | | '''Sat July 27th - Mon 29th (incl.)''' |
|- | |- | ||
! Location | ! Location | ||
| − | | | + | | Cahill room 370, H323 call ins from AEI, TAT. ([[ET_Workshop_Summer_2013_(Developers_Workshop)#h323 Instructions|call in instructions]]) |
|- | |- | ||
! Contacts | ! Contacts | ||
| Line 31: | Line 29: | ||
At any terminal at LAX, you will find departure platforms for all of the above transportation systems just outside the terminals. | At any terminal at LAX, you will find departure platforms for all of the above transportation systems just outside the terminals. | ||
| − | The FlyAway Bus + Metro is usually very convenient. To get on it, just wait by the blue sign that says "FlyAway Busses". Wait until a FlyAway Bus with sign "Union Station" arrives (it departs every ~30min). Jump on it and go until Union Station (its only and final destination). You pay $7 upon arrival (they have change if you don't exactly have $7). At Union Station, go to the platform that says "Metro Gold Line". Pay $1.50 at the ticket machine (one way ticket). Take the Metro Gold Line (direction "Sierra Madre Station") and exit at "Lake Station" (Caltech or Vagabond Inn) or "Allen Station" (best for the Saga Motor Hotel). Check out google maps for further info. Here's a | + | The FlyAway Bus + Metro is usually very convenient. To get on it, just wait by the blue sign that says "FlyAway Busses". Wait until a FlyAway Bus with sign "Union Station" arrives (it departs every ~30min). Jump on it and go until Union Station (its only and final destination). You pay $7 upon arrival (they have change if you don't exactly have $7). At Union Station, go to the platform that says "Metro Gold Line". Pay $1.50 at the ticket machine (one way ticket). Take the Metro Gold Line (direction "Sierra Madre Station") and exit at "Lake Station" (Caltech or Vagabond Inn) or "Allen Station" (best for the Saga Motor Hotel). Check out google maps for further info. Here's a link to Metro Gold Line, including a link to this trip planer (it also knows about FlyAway busses). |
|- | |- | ||
! Transportation - Workshop | ! Transportation - Workshop | ||
| − | | walking distance, [https://maps.google. | + | | walking distance, [https://maps.google.com/maps?saddr=Vagabond+Inn+Executive+Pasadena,+1203+East+Colorado+Boulevard,+Pasadena,+CA+91106,+United+States&daddr=E+California+Blvd&hl=en&ll=34.141006,-118.123884&spn=0.02156,0.033002&sll=34.136797,-118.124335&sspn=0.010781,0.016501&geocode=FSAHCQIdjIv1-CmXX-XzWsPCgDEGkCOqwhMTwA%3BFWjfCAId8In1-A&oq=1200+E+Calif+Blvd,+Pasadena,+CA,+United+States&dirflg=w&mra=dme&mrsp=1&sz=16&t=m&z=15 directions]. |
|- | |- | ||
! Food options | ! Food options | ||
| Line 41: | Line 39: | ||
|} | |} | ||
| − | == | + | ===== h323 Instructions ===== |
| − | + | The room's IP address is 131.215.195.4, it should accept calls. '''PLEASE DO NOT TEST''' this right now since the room is often in use for group meetings etc. Instead check the [http://mrbs.astro.caltech.edu/week.php?year=2013&month=07&day=21&area=8&room=20 room booking page] and send an email to [mailto:rhaas@tapir.caltech.edu Roland Haas] who will operate to room for a test. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| + | We have had best success so far by connecting from another conference room system (Tandberg, Polycom seem to work). Failing that it is possible to use Ekiga for Linux (call h323:131.215.195.4). However we found that the versions of Ekiga in Ubuntu and Debian require specific settings to work. In particular the only audio/video codec combination that seems to work is G722 for audio and H261 for video. Ekiga lets you enable/disable codecs in its options dialog. We do not know of a working client for OSX or Windows at this point (XMeeting claims to be H323 client for OSX). | ||
==== Participants ==== | ==== Participants ==== | ||
| − | for the | + | for the developers' workshop (July 27th to 29th): |
{|class="wikitable" border="2" cellpadding="2" | {|class="wikitable" border="2" cellpadding="2" | ||
| Line 109: | Line 58: | ||
| Peter Diener || Louisiana State University || Vagabond | | Peter Diener || Louisiana State University || Vagabond | ||
|- | |- | ||
| − | | Josh Faber || RIT || | + | | Josh Faber || RIT || Saga |
|- | |- | ||
| Roland Haas || Caltech || local | | Roland Haas || Caltech || local | ||
| Line 120: | Line 69: | ||
|- | |- | ||
| Ian Hinder || Albert Einstein Institute || Remote | | Ian Hinder || Albert Einstein Institute || Remote | ||
| + | |- | ||
| + | | Tanja Bode || Universität Tübingen || Remote | ||
|- | |- | ||
|} | |} | ||
| Line 125: | Line 76: | ||
== In-Workshop Details == | == In-Workshop Details == | ||
| + | Want to have a relatively large list of possible topics then select which ones to actually work on. | ||
| + | |||
| + | Ideas for topics: | ||
| + | |||
| + | * Vectorization | ||
| + | * AMR grid setup | ||
| + | * AMR scaling | ||
| + | * IO scaling | ||
| + | * Data-driven scheduler | ||
| + | * Cleanup of GRHydro | ||
| + | * Automated performance tests to verify that changes are actually beneficial | ||
| + | * MHD development: | ||
| + | ** Refluxing | ||
| + | ** A-field MHD | ||
| + | * David Rideout offered to give a presentation | ||
| + | * McLachlan: Erik and Ian want to make several changes to the code: | ||
| + | ** Allow arguments to be passed via scripts to generate multiple thorns in parallel, rather than sequentially | ||
| + | ** Design of more high level APIs to streamline declaration of new objects, e.g. tensors | ||
| + | ** Introduce master calculation for BSSN variables, and use filtering to isolate necessary code to be called in a specific run | ||
| + | ** Enable making parameters available at either runtime or Kranc time for greater flexibility | ||
| + | ** Additional optimizations | ||
| − | |||
=== Schedule === | === Schedule === | ||
| Line 136: | Line 107: | ||
| Presenter / Responsible | | Presenter / Responsible | ||
|- | |- | ||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
|} | |} | ||
| − | === | + | === Setup === |
| + | |||
| + | * Wireless access is through the (dedicated) CGWAS network. | ||
| − | + | == Notes == | |
| − | + | ||
| − | + | === Topics === | |
| − | + | Todo for sure: | |
| − | + | * DR talk | |
| − | + | * Chemora | |
| − | + | ||
| − | + | list of votes on topics: | |
| − | + | * (3) MHD, A, s | |
| − | + | * (1) GRHydro cleanup | |
| − | + | * (3) GRHydro vectorization | |
| − | + | * (2) McLachlan | |
| − | + | * (3) scaling | |
| − | + | * (2) benchmarks | |
| − | + | ||
| − | + | === Chemora === | |
| − | + | * Ian: Paper recently added to arxiv: http://arxiv.org/abs/1307.6488 | |
| − | + | * got funded (LSU, GaTech, NCSA, [Beijing]) | |
| − | + | ** Money should benefit the whole Chemora group. Possible get together in Atlanta as Zhihui Du, our Chinese collaborator, will be there for the next year or so. Other venues are possible. | |
| − | + | * "PDE" compiler | |
| − | + | ** Kranc, maybe with Hydro -> CaKernel | |
| − | + | ** Need more language features in the PDE language | |
| − | + | * Use of accelerators (and CPUs) | |
| − | + | ** OpenCL (for all CUs) - Want this to make sure we can run on any new machine. | |
| − | + | ** CUDA - Want to be able to use this. CaKernel can do CUDA or OpenCL. | |
| − | + | * Machine profile | |
| − | + | ** hwloc | |
| − | + | ** PAPI (now also works on Intel MICs) - dynamic feedback for LoopControl or other things. | |
| − | + | * Production use | |
| − | + | * Not just for exascale | |
| − | + | ** The concern is that a wide variety of big machines be enabled, too many proposals were targeting exascale. | |
| − | + | * Intermediate Black Hole Binary Science Driver - Gatech primarily responsible. However, the Chinese side recently put in a proposal to Titan for this. Getting benefits in GRHydro may, however, be more important. | |
| − | + | * Load Balancing - We'll need Peter's load balancer. Using AMPI with Cactus was also promised. | |
| − | + | * First steps | |
| − | + | ** Get the code working again | |
| − | + | ** Run on one node | |
| − | + | ** Focus on Supermike 2 | |
| − | + | * Mailing list (hackathon) | |
| − | + | * Website | |
| − | + | ||
| − | + | === Scaling === | |
| − | + | * Quote of the day: Christian Ott: "'''Have you solved the scaling problem yet?'''" | |
| − | + | * Carpet switched to new internal representation of the box-datastructure, improving scaling of modify-operations of this (initial setup, regridding) | |
| − | + | * Since Hydro isn't vectorized, we loose performance on vector machines, unsurprisingly. | |
| − | + | ||
| − | + | === David Rideout's Talk === | |
| − | + | * Directions for Cactus HPC Framework | |
| − | + | * Tension between generic nature of framework vs. scientific results | |
| − | + | * Tangent discussion: need for elliptic solver to increase userbase, but hard | |
| − | + | **More on the tangent: whether to use packages or write from scratch | |
| − | + | ** Ian: See http://arxiv.org/abs/1305.5576 which will be free software soon. This is a generic multigrid elliptic solver using Carpet. | |
| − | + | * Big Data: esp. phone data is important current research topic | |
| − | + | * Cosmological simulations can be modeled using similar techniques as networks | |
| − | + | * Relevance to Cactus | |
| − | + | ** Network data as discrete spacetime model | |
| − | + | ** Discrete quantum gravity | |
| − | + | ** Unstructured Meshes | |
| − | + | * Changes to Cactus to incorporate different grid structures and models may be quite significant, limited experience for current maintainers | |
| − | + | ||
| − | + | === Erik Schnetter's Paper === | |
| − | + | ||
| − | + | * Performance and Optimization Abstractions for Large Scale Heterogeneous Systems in the Cactus/Chemora Framework: [[Media:xscale2013.pdf]] | |
| − | + | ||
| − | + | === Staggered A-field Evolution === | |
| − | + | ||
| − | + | In the staggered system, B-fields live on faces (gets its own gridfunction) | |
| − | + | A-field lives on edges (centered versions get a new variable) | |
| − | + | E-field lives on edges, but may be stored locally (?) | |
| − | + | reconstruction needs to propagate some quantities, like velocity from centers to edges. | |
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | |||
| − | + | * Changes were made to interfaces and params, and committed to Zelmani (may change as needed) | |
| − | * | + | * Tasks to work on |
| − | * | + | ** Prolongation/restriction/carpetLib operators -- Philipp |
| + | ** HLLE for the Avec, being careful to consider which variables live where -- Philipp | ||
| + | ** Reconstruction for Bvec, velocities (doubly so) at edges -- Frank/Tanja | ||
| + | ** Calc Update, Source Terms, averaged B field for fluid fluxes -- Josh | ||
| + | ** Documentation of the method -- Josh/??? | ||
Latest revision as of 13:07, 29 July 2013
ET Workshop Summer 2013 (Developers Workshop
Developers' workshop
Spacetime Coordinates and Contact Information
| Dates | Sat July 27th - Mon 29th (incl.) |
|---|---|
| Location | Cahill room 370, H323 call ins from AEI, TAT. (call in instructions) |
| Contacts | Workshop email list: workshop@einsteintoolkit.org, emergencies: Frank Löffler (knarf@cct.lsu.edu, +1-225-202-1752) or Roland Haas (rhaas@tapir.caltech.edu, +1-404-386-3657) |
| Hotel | Saga Motor Inn, 1633 E. Colorado Blvd., Pasadena, CA 91106, Preferred reservation line, Phone (626) 795-0431, Fax (626) 792-0559, http://www.thesagamotorhotel.com, $65.00 double bed, one person, $70.00 double bed, two people, Caltech rate (mention when calling)
Vagabond Inn, Pasadena, 1203 East Colorado Blvd., Pasadena, CA 91106, Phone (626) 449-3170, http://www.vagabondinn.com, $79.00 double bed (one or two persons), rate on website differs, best to call and mention Caltech. |
| Transportation - Hotel | There are many possible ways of getting from the airport to Pasadena:
At any terminal at LAX, you will find departure platforms for all of the above transportation systems just outside the terminals. The FlyAway Bus + Metro is usually very convenient. To get on it, just wait by the blue sign that says "FlyAway Busses". Wait until a FlyAway Bus with sign "Union Station" arrives (it departs every ~30min). Jump on it and go until Union Station (its only and final destination). You pay $7 upon arrival (they have change if you don't exactly have $7). At Union Station, go to the platform that says "Metro Gold Line". Pay $1.50 at the ticket machine (one way ticket). Take the Metro Gold Line (direction "Sierra Madre Station") and exit at "Lake Station" (Caltech or Vagabond Inn) or "Allen Station" (best for the Saga Motor Hotel). Check out google maps for further info. Here's a link to Metro Gold Line, including a link to this trip planer (it also knows about FlyAway busses). |
| Transportation - Workshop | walking distance, directions. |
| Food options | Chandler's on campus, Ernie's food truck (times), restaurants on Lake, anything on Colorado. |
h323 Instructions
The room's IP address is 131.215.195.4, it should accept calls. PLEASE DO NOT TEST this right now since the room is often in use for group meetings etc. Instead check the room booking page and send an email to Roland Haas who will operate to room for a test.
We have had best success so far by connecting from another conference room system (Tandberg, Polycom seem to work). Failing that it is possible to use Ekiga for Linux (call h323:131.215.195.4). However we found that the versions of Ekiga in Ubuntu and Debian require specific settings to work. In particular the only audio/video codec combination that seems to work is G722 for audio and H261 for video. Ekiga lets you enable/disable codecs in its options dialog. We do not know of a working client for OSX or Windows at this point (XMeeting claims to be H323 client for OSX).
Participants
for the developers' workshop (July 27th to 29th):
| Name | Affiliation | Hotel |
|---|---|---|
| Steven Brandt | Louisiana State University | Vagabond |
| Peter Diener | Louisiana State University | Vagabond |
| Josh Faber | RIT | Saga |
| Roland Haas | Caltech | local |
| Frank Löffler | Louisiana State University | Vagabond |
| Philipp Mösta | Caltech | local |
| Christian Ott | Caltech | local |
| Ian Hinder | Albert Einstein Institute | Remote |
| Tanja Bode | Universität Tübingen | Remote |
In-Workshop Details
Want to have a relatively large list of possible topics then select which ones to actually work on.
Ideas for topics:
- Vectorization
- AMR grid setup
- AMR scaling
- IO scaling
- Data-driven scheduler
- Cleanup of GRHydro
- Automated performance tests to verify that changes are actually beneficial
- MHD development:
- Refluxing
- A-field MHD
- David Rideout offered to give a presentation
- McLachlan: Erik and Ian want to make several changes to the code:
- Allow arguments to be passed via scripts to generate multiple thorns in parallel, rather than sequentially
- Design of more high level APIs to streamline declaration of new objects, e.g. tensors
- Introduce master calculation for BSSN variables, and use filtering to isolate necessary code to be called in a specific run
- Enable making parameters available at either runtime or Kranc time for greater flexibility
- Additional optimizations
Schedule
| Day | Time | Activity | Presenter / Responsible |
Setup
- Wireless access is through the (dedicated) CGWAS network.
Notes
Topics
Todo for sure:
- DR talk
- Chemora
list of votes on topics:
- (3) MHD, A, s
- (1) GRHydro cleanup
- (3) GRHydro vectorization
- (2) McLachlan
- (3) scaling
- (2) benchmarks
Chemora
- Ian: Paper recently added to arxiv: http://arxiv.org/abs/1307.6488
- got funded (LSU, GaTech, NCSA, [Beijing])
- Money should benefit the whole Chemora group. Possible get together in Atlanta as Zhihui Du, our Chinese collaborator, will be there for the next year or so. Other venues are possible.
- "PDE" compiler
- Kranc, maybe with Hydro -> CaKernel
- Need more language features in the PDE language
- Use of accelerators (and CPUs)
- OpenCL (for all CUs) - Want this to make sure we can run on any new machine.
- CUDA - Want to be able to use this. CaKernel can do CUDA or OpenCL.
- Machine profile
- hwloc
- PAPI (now also works on Intel MICs) - dynamic feedback for LoopControl or other things.
- Production use
- Not just for exascale
- The concern is that a wide variety of big machines be enabled, too many proposals were targeting exascale.
- Intermediate Black Hole Binary Science Driver - Gatech primarily responsible. However, the Chinese side recently put in a proposal to Titan for this. Getting benefits in GRHydro may, however, be more important.
- Load Balancing - We'll need Peter's load balancer. Using AMPI with Cactus was also promised.
- First steps
- Get the code working again
- Run on one node
- Focus on Supermike 2
- Mailing list (hackathon)
- Website
Scaling
- Quote of the day: Christian Ott: "Have you solved the scaling problem yet?"
- Carpet switched to new internal representation of the box-datastructure, improving scaling of modify-operations of this (initial setup, regridding)
- Since Hydro isn't vectorized, we loose performance on vector machines, unsurprisingly.
David Rideout's Talk
- Directions for Cactus HPC Framework
- Tension between generic nature of framework vs. scientific results
- Tangent discussion: need for elliptic solver to increase userbase, but hard
- More on the tangent: whether to use packages or write from scratch
- Ian: See http://arxiv.org/abs/1305.5576 which will be free software soon. This is a generic multigrid elliptic solver using Carpet.
- Big Data: esp. phone data is important current research topic
- Cosmological simulations can be modeled using similar techniques as networks
- Relevance to Cactus
- Network data as discrete spacetime model
- Discrete quantum gravity
- Unstructured Meshes
- Changes to Cactus to incorporate different grid structures and models may be quite significant, limited experience for current maintainers
Erik Schnetter's Paper
- Performance and Optimization Abstractions for Large Scale Heterogeneous Systems in the Cactus/Chemora Framework: Media:xscale2013.pdf
Staggered A-field Evolution
In the staggered system, B-fields live on faces (gets its own gridfunction) A-field lives on edges (centered versions get a new variable) E-field lives on edges, but may be stored locally (?) reconstruction needs to propagate some quantities, like velocity from centers to edges.
- Changes were made to interfaces and params, and committed to Zelmani (may change as needed)
- Tasks to work on
- Prolongation/restriction/carpetLib operators -- Philipp
- HLLE for the Avec, being careful to consider which variables live where -- Philipp
- Reconstruction for Bvec, velocities (doubly so) at edges -- Frank/Tanja
- Calc Update, Source Terms, averaged B field for fluid fluxes -- Josh
- Documentation of the method -- Josh/???
