2017 MHD Workshop

Time and Agenda

Monday
2:00 — 2:10: Welcome
2:10 — 2:35: Philipp to talk on flux CT
2:35 — 3:00: Daniel to talk on vector potential and MHD con2prim
3:00 — 3:25: (Propose switch with Christian's talk) David to talk on DG and Zelmani M1
3:25 — 3:55: Coffee break
3:55 — 4:20: (Propose switch with David's talk) Christian to talk on M1 work/results
4:20 — 4:45: Erik: FunHPC — current state, toward AMR, toward KNL
4:45 — 5:10: Roland: HydroFunToy — code design and capability
5:10 — 5:45: Closing discussion and roadmap for the week

Tuesday to Thursday
9:00 am till late: “workshop” -> work on topics identified on Monday

Friday
9:00 — 11:00: Wrap it up
11:00 — 12:30: Summary of the workshop and defining future directions/coordinate future work
12:30: lunch/end of workshop

• You are free to arrive earlier on Monday and leave later on Friday, we have the room booked for the entire day.
• Let me know if this tentative agenda conflicts with any of your travel plans, we can reschedule things.

Talks

• The talks on Monday afternoon are meant to be informal kick-off talks in order to start the discussion in a well defined way and to identify the topics/aspects that we want to work on in the following days (or that we want to keep in mind while working on other aspects). I’d suggest that we keep them fairly short (no more than ~20-25 mins including some discussion), just trying to provide a basic introduction that brings everyone onto the same page, addressing the following basic questions like:
  • What is the analytic formulation (basic intro to the equations — if applicable)?
  • What is the current status, or what have people been working on so far?
  • What are the short & long-term goals? What do we need to implement/what needs to be done to move forward?
  • What do you think could be the specific goals for the week?

• Please check and update the title of your talk you volunteered for (see above)

• Feel free to add some notes/thoughts/ideas regarding your talk on the wiki

Knights Landing on Stampede

• Stampede manual: https://portal.tacc.utexas.edu/user-guides/stampede
• Login node: login-knl1.stampede.tacc.utexas.edu
• option list, submission scripts, and environment settings for both icc and gcc are available for batchtools: https://bitbucket.org/dradice/batchtools

flux-CT implementation progress

• Identified which equations to solve and that there is indeed a Riemann problem to solve for the electric field
• This is not a Riemann problem for the state vector (which is the magnetic field), but the electric field is used to update the magnetic field
• The Riemann problem for the electric field requires reconstruction of the velocity and magnetic field to the cell edge, requiring a double reconstruction for the velocity
• Started implementation for that and bookkeeping is going to be tedious but otherwise implementation should be straightforward
• Need to come up with good test cases to make sure this is implemented correctly

FunHPC parallelization progress

• Continued to make FunHPC work, as alternative to OpenMP
• Improve performance by taking cache alignment into account
  • Many internal modifications to Cactus and Carpet
• Benchmarking
  • Need to measure performance details automatically, make available to users

Reduced width operators (ENOG2)

• implementation finished and tested with SyncG2Test to verify that indeed only the inner 2 ghost points are synchronized and only 2*2=4 points are prolongated
• currently no overall speedup of a M1 simulation observed
• requires more careful testing where time is spend during SYNC

New Con2Prim Interface for GRHydro

The goal of the Next Gen Con2Prim (Con2PrimNG) interface for GRHydro is to allow for customized Con2Prim routines provided by the user in a separate thorn. To enable this, GRHydro can now call an aliased function for Con2Prim provided by a dedicated user-provided Thorn.

The changes to GRHydro are as follows:

• In interface.ccl, introduced relevant ccl code so that GRHydro can use a function called GRHydro_Con2PrimNG for Con2Prim.
• In param.ccl, introduced boolean 'GRHydro_use_Con2PrimNG'. It defaults to 'no', but if set to 'yes', GRHydro will attempt to call the aliased function GRHydro_Con2PrimNG for Con2Prim. Also removed two parameters that were used for the BBH+Gas project and that were recently introduced and have not been pushed upstream.
• In schedule.ccl, commented out all calls to Con2Prim routines and replaced them with call to a new wrapper function defined in src/GRHydro_Con2Prim.cc.
• Introduced new C2P wrapper routine that is called from the schedule. This is in GRHydro/src/GRHydro_Con2Prim.cc. This routine provides full backward compatibility and calls previous C and F90 C2P routines if GRHydro_use_Con2PrimNG set to 'no' and no function GRHydro_Con2PrimNG is provided through the aliased function interface.
• Cleaned up GRHydro/src/GRHydro_Con2Prim.F90 and removed code specific to the BBH+Gas project. The implementation in GRHydro_Con2Prim.F90 should not be back to the generic one in the ET version of GRHydro.
• Added sanity check code in GRHydro_ParamCheck.F90 that makes sure that no parameter-level disasters can happen.

As an example for an external Con2Prim Thorn that provides GRHydro_Con2PrimNG, Christian wrote Zelmani_Con2PrimNG_BBHGas, which implements a variant of the vanilla GRHydro Con2Prim routine and makes it available to GRHydro via the aliased function interface. Note that since the backend C2P routine is F90, an additional wrapper is required that massages the passed arguments in such a way so that the backend F90 routine can be called with CCTK_ARGUMENTS. In C or C++ such a wrapper would not be necessary.

Implementation of vector potential into GRHydro

Quick summary:

• Implemented staggered vector potential formalism according to Etienne et al. 2012 and the IllinoisGRMHD code
• This is contained in the Zelmani branch vector_potential
• Essentially isolated, rewrote or modified relevant code parts from IllinoisGRMHD and included them into GRHydro
• Removed old (dead) vector potential parts in GRHydro and made several changes to GRHydro core routines to make it work with staggered Avec and Bvec fields
• Currently compiles and only works with the cxx files/version of GRHydro
• Added initial data routine to GRHydro_InitData to initialize magnetic field configurations for the staggered Avec and Bvec fields

Next steps:

• First test runs for debugging
• Modify GRHydro's reconstruction routines to also reconstruct staggered B and velocity fields for computation of RHS of staggered Avec evolution (currently using a modified routine from IllinoisGRMHD)
• Modification of GRHydro's reconstruction routines to be coordinated with Philipp's work on new staggered CT scheme, as similar infrastructure required