Workshop in New York City and at Columbia University.

== Chelsea Venue ==

* On the first afternoon, we'll meet off-campus to get the workshop started.
* We will start at 14:00.
* We have a conference room for 8 booked under Christian D. Ott, California Institute of Technology at Select Office Suites Chelsea 116 W 23rd St, New York, NY 10011 https://goo.gl/maps/WDBNQpwadfE2 From Columbia, the best way to get there is to take Subway Line 1 to 23rd Street Station.

== Columbia Venue==

* The workshop will take place in the Conference Room of the Center for Theoretical Physics at Columbia. The Conference Room (Room 907) is located on the 9th floor of the Physics building (Pupin Laboratories).
* Street address: 538 West 120th Street, New York, NY 10027
* Note that the main entrance is on the Campus Level (5th floor, south side) as indicated here:
[https://www.google.com/maps/place/40°48'35.9%22N+73°57'41.2%22W/@40.809983,-73.9619872,19z/data=!3m1!4b1!4m5!3m4!1s0x0:0x0!8m2!3d40.809982!4d-73.96144 map]
* You can enter Campus at 116th/Broadway. There is also the possibility of entering Pupin Hall from 120th/Broadway. In this case, enter through the North West Corner Building and take the stairs/escalators up to the Campus level, exit the building and enter Pupin through the main entrance.
* The conference room has a board and a big presentation screen -> There is the possibility of giving a black board talk or a slide presentation, or a combination of both.

==Time and Agenda==

Monday 
2:00 — 2:10: Welcome 
2:10 — 2:35: Philipp to talk on [[2017_MHD_Workshop/fluxCT|flux CT]] 
2:35 — 3:00: Daniel to talk on [[2017_MHD_Workshop/vector_potential|vector potential]] and [[2017_MHD_Workshop/MHD_con2prim|MHD con2prim]] 
3:00 — 3:25: ''(Propose switch with Christian's talk)'' David to talk on DG and [[2017_MHD_Workshop/Zelmani_M1|Zelmani M1]] 
3:25 — 3:55: Coffee break 
3:55 — 4:20: ''(Propose switch with David's talk)'' Christian to talk on [[2017_MHD_Workshop/Zelmani_M1|M1 work]]/results 
4:20 — 4:45: Erik: [[2017_MHD_Workshop/FunHPC|FunHPC]] — current state, toward AMR, toward KNL 
4:45 — 5:10: Roland: [[2017_MHD_Workshop/FunHPC|HydroFunToy]] — code design and capability 
5:10 — 5:45: Closing discussion and roadmap for the week

Tuesday to Thursday 
9:00 am till late: “workshop” -> work on topics identified on Monday 

Friday 
9:00 — 11:00: Wrap it up 
11:00 — 12:30: Summary of the workshop and defining future directions/coordinate future work 
12:30: lunch/end of workshop 

* You are free to arrive earlier on Monday and leave later on Friday, we have the room booked for the entire day.
* Let me know if this tentative agenda conflicts with any of your travel plans, we can reschedule things.

==Talks==

* The talks on Monday afternoon are meant to be informal kick-off talks in order to start the discussion in a well defined way and to identify the topics/aspects that we want to work on in the following days (or that we want to keep in mind while working on other aspects). I’d suggest that we keep them fairly short (no more than ~20-25 mins including some discussion), just trying to provide a basic introduction that brings everyone onto the same page, addressing the following basic questions like:
** What is the analytic formulation (basic intro to the equations — if applicable)?
** What is the current status, or what have people been working on so far?
** What are the short & long-term goals? What do we need to implement/what needs to be done to move forward?
** What do you think could be the specific goals for the week?

* Please check and update the title of your talk you volunteered for (see above)

* Feel free to add some notes/thoughts/ideas regarding your talk on the wiki

== Knights Landing on Stampede ==

* Stampede manual: https://portal.tacc.utexas.edu/user-guides/stampede
* Login node: login-knl1.stampede.tacc.utexas.edu
* option list (beta): https://dl.dropboxusercontent.com/u/17077801/stampede-knl.cfg
* environment (beta): https://dl.dropboxusercontent.com/u/17077801/stampede-knl.env
* submission script (beta): https://dl.dropboxusercontent.com/u/17077801/stampede-knl.sub

David Radice's simfactory replacement: https://bitbucket.org/dradice/batchtools

== flux-CT implementation progress ==

* Identified which equations to solve and that there is indeed a Riemann problem to solve for the electric field
* This is not a Riemann problem for the state vector (which is the magnetic field), but the electric field is used to update the magnetic field
* The Riemann problem for the electric field requires reconstruction of the velocity and magnetic field to the cell edge, requiring a double reconstruction for the velocity
* Started implementation for that and bookkeeping is going to be tedious but otherwise implementation should be straightforward
* Need to come up with good test cases to make sure this is implemented correctly

==FunHPC parallelization progress==

* Continued to make FunHPC work, as alternative to OpenMP
* Improve performance by taking cache alignment into account
* * Many internal modifications to Cactus and Carpet
* Benchmarking
* * Need to measure performance details automatically, make available to users

2017 MHD Workshop

2017-02-03T11:04:39Z

Eschnett: /* flux-CT implementation progress */

2017 MHD Workshop

2017-01-23T12:45:53Z

Eschnett:

workshop at Columbia.

Meeting minutes (delete once done):
* Daniel and Philipp have discussed a bit what they want to cover and
prepared a draft and to use a wiki (suggest ET wiki, all agree)
** a page is here:
https://docs.einsteintoolkit.org/et-docs/2017_MHD_Workshop

* Daniel will provide some local information
* interest in radiation transport and its current limitations
* interest to get an overview about HydroFunHPC code (design idea, iintention) and FunHPC in general
** Currently HydroFunHPC has all the pieces but fails (likely in con2prim) with NaNs on the surface of a TOV star.
** ultimately this requires integration with Carpet to overlap more communication and computation

* plan:
** kickoff talks on Monday
** Philipp to talk about CT
** Daniel to talk about vector potential method and MHD con2prim
** Christian and David to talk about M1
** Erik and Roland to talk about (Hydro)FunHPC

==Kick-Off Talks==

* Erik: FunHPC -- current state, towards AMR, towards KNL

2017 MHD Workshop

2017-01-23T12:45:37Z

Eschnett:

workshop at Columbia.

Meeting minutes (delete once done):
* Daniel and Philipp have discussed a bit what they want to cover and
prepared a draft and to use a wiki (suggest ET wiki, all agree)
** a page is here:
https://docs.einsteintoolkit.org/et-docs/2017_MHD_Workshop

* Daniel will provide some local information
* interest in radiation transport and its current limitations
* interest to get an overview about HydroFunHPC code (design idea, iintention) and FunHPC in general
** Currently HydroFunHPC has all the pieces but fails (likely in con2prim) with NaNs on the surface of a TOV star.
** ultimately this requires integration with Carpet to overlap more communication and computation

* plan:
** kickoff talks on Monday
** Philipp to talk about CT
** Daniel to talk about vector potential method and MHD con2prim
** Christian and David to talk about M1
** Erik and Roland to talk about (Hydro)FunHPC

## Kick-Off Talks

* Erik: FunHPC -- current state, towards AMR, towards KNL

2017 MHD Workshop

2017-01-23T12:44:38Z

Eschnett:

Remote Mini-Workshop Series

2016-12-14T12:23:16Z

Eschnett: /* To Do */

Quite a few interesting mini-projects are being undertaken at the moment. It is worthwhile to advertise these to the larger community to invite participation. In our weekly calls we decided that we should set aside a few hours or half a day for one of these. I now suggest that we turn this into a mini-series, where we pick from the list below until we run out of interest. Maybe this will keep us busy until Christmas.

We picked Wednesday 9:00 EST as meeting time. We'll meet on Google Hangout (probably), details TBA here.

# Spack: installing external package https://github.com/LLNL/spack [Erik]
# SimulationIO: a new file format that's easy to read https://github.com/eschnett/SimulationIO
# FunHPC (multi-threading with futures): overview https://bitbucket.org/eschnett/funhpc.cxx [Erik, Christian, Ian]
# FunHPC (multi-threading with futures): shoehorning this into Cactus [Erik, Christian, Ian]
# StencilOps: more efficient finite differencing stencils in Kranc [Ian]
# DG: Jonah and my new DG formulation that can replace FD methods https://arxiv.org/abs/1604.00075 [Federico]
# The "distribute" script: testing the Einstein Toolkit on HPC systems
# Towards a Kranc implementation of a hydro formulation [Ian, Federico]

If you are interested in one of these topics, then add your name in square brackets after the topic.

If you are interested in presenting a topic yourself, then add a new item to the list.

==Mini-Workshop #1: Wed, Dec 7, 2016, 9:00 EST==

Topic: FunHPC (multi-threading with futures): overview https://bitbucket.org/eschnett/funhpc.cxx

Venue: Google Hangouts https://hangouts.google.com/call/jjkffrrvmnbhrooiyjxhfeb2ume

Agenda:
* FunHPC design overview
* Comparison to OpenMP
* CPU vs. memory performance
* Cache and multi-threading, loop tiling
* How to parallelize an application via FunHPC
* Building and installing
* Examples
* Benchmarks

===Building and Installing===

FunHPC is available on BitBucket https://bitbucket.org/eschnett/funhpc.cxx . It requires several other packages to be installed as well, namely
* Cereal: Serializing C++ objects http://uscilab.github.io/cereal
* hwloc: Determining the hardware (core, cache) layout http://www.open-mpi.org/projects/hwloc
* jemalloc: Fast multi-threaded memory manager (malloc replacement) http://www.canonware.com/jemalloc
* OpenMPI: FunHPC prefers this MPI library http://www.open-mpi.org
* Qthreads: Fine-grained multi-threading (providing a C interface) http://www.cs.sandia.gov/qthreads

To install FunHPC from scratch, you need to install these other libraries first, and then edit FunHPC's Makefile. Google Test is also required, but will be downloaded automatically. Apologies for this unprofessional setup. In the future, FunHPC should be converted to use cmake, and Google Test should be packages as part of it.

The Cereal package requires a patch. This patch makes it distinguish between regular pointers and function pointers. Regular pointers cannot be serialized since it is unclear whether they are valid, and if so, how the target should be allocated or freed. Function pointers, however, can be serialized -- we assume they point to functions, which are constants, so that no memory management issues arise. You need to apply the following patch:

--- old/include/cereal/types/common.hpp
+++ new/include/cereal/types/common.hpp
@@ -106,14 +106,16 @@
t = reinterpret_cast<typename common_detail::is_enum<T>::type const &>( value );
}

+#ifndef CEREAL_ENABLE_RAW_POINTER_SERIALIZATION
//! Serialization for raw pointers
/*! This exists only to throw a static_assert to let users know we don't support raw pointers. */
template <class Archive, class T> inline
void CEREAL_SERIALIZE_FUNCTION_NAME( Archive &, T * & )
{
static_assert(cereal::traits::detail::delay_static_assert<T>::value,
"Cereal does not support serializing raw pointers - please use a smart pointer");
}
+#endif

//! Serialization for C style arrays
template <class Archive, class T> inline

When you "make", you need to pass certain environment variables:
* CEREAL_DIR (have to set in Makefile)
* HWLOC_DIR
* JEMALLOC_DIR
* QTHREADS_DIR
* CXX
* MPICXX
* MPIRUN

For example:
make CEREAL_DIR=... HWLOC_DIR=... JEMALLOC_DIR=... QTHREADS_DIR=... CXX=c++ MPICXX=mpicxx MPIRUN=mpirun

I have installed FunHPC and all its dependencies on Wheeler (Caltech) into the directory /home/eschnett/src/spack-view . This includes a recent version of GCC that was used to build these libraries. If you want to use this, then I highly recommend using this version of GCC as well as all the other software installed in this directory (e.g. HDF5, PAPI, and many more) instead of combining these with system libraries.

As a side note, Roland Haas says that the Simfactory configuration for Wheeler is using this directory. This is not really relevant yet since we won't be using Cactus in the beginning.

===Running FunHPC Applications===

FunHPC is an MPI application, but we are not interested in using MPI today. We might still need to use mpirun, but only in a trivial way.

Qthreads etc. use environment variables to change certain settings. Some settings are necessary to prevent problems. These "problems" are usually resource exhaustion (e.g. not enough stack space), which Unix helpfully all translates into "Segmentation fault". I am usually setting these environment variables:

export QTHREAD_NUM_SHEPHERDS="${nshep}"
export QTHREAD_NUM_WORKERS_PER_SHEPHERD="${nwork}"
export QTHREAD_STACK_SIZE=8388608 # Byte
export QTHREAD_GUARD_PAGES=0 # 0, 1
export QTHREAD_INFO=1

Here "nshep" is the number of sockets (aka NUMA nodes), and "nwork" the number of cores per socket. You can find these e.g. via "hwloc-info". On Wheeler:

$ ~/src/spack-view/bin/hwloc-info
depth 0: 1 Machine (type #1)
depth 1: 2 NUMANode (type #2)
depth 2: 2 Package (type #3)
depth 3: 2 L3Cache (type #4)
depth 4: 24 L2Cache (type #4)
depth 5: 24 L1dCache (type #4)
depth 6: 24 L1iCache (type #4)
depth 7: 24 Core (type #5)
depth 8: 24 PU (type #6)

Thus I choose "nshep=2" and "nwork=12" on Wheeler.

By default, Qthreads chooses a rather small stack size of 8 kByte per thread. If a thread uses more stack space, random memory will be overwritten. You can enable guard pages, which is good for debugging. This will catch many cases where the stack overflows. Finally, Qthreads can produce info output at startup that might be helpful.

On Wheeler:
~eschnett/src/spack-view/bin/mpirun -np 1 -x QTHREAD_NUM_SHEPHERDS=2 -x QTHREAD_NUM_WORKERS_PER_SHEPHERD=12 -x QTHREAD_STACK_SIZE=1000000 ~eschnett/src/spack-view/bin/fibonacci

===Loop Example===

Let us look at a simple loop. We are going to parallelize it once with OpenMP, and once with FunHPC.

#include <funhpc/async.hpp>
#include <funhpc/main.hpp>

#include <algorithm>
#include <cassert>
#include <vector>

// Synchronize the ghost zones (the outermost points in each direction)
void sync(double *y, int n) {
// we just assume periodic boundaries
assert(n >= 2);
y[0] = y[n - 2];
y[n - 1] = y[1];
}

// A basic loop, calculating a first derivative
void vdiff(double *y, const double *x, int n) {
for (int i = 1; i < n - 1; ++i) {
y[i] = (x[i + 1] - x[i - 1]) / 2;
}
sync(y, n);
}

// The same loop, parallelized via OpenMP: The number of iterations is
// split over the available number of cores (e.g. 12 on a NUMA node of
// Wheeler). The disadvantages of this are:
// - There is a implicit barrier at the end of the loop, so that the
// sync afterwards cannot overlap with the loop
// - A single thread might handle too much work, overflowing the cache
// - A single thread might not have enough work, so that the thread
// management overhead is too large
void vdiff_openmp(double *y, const double *x, int n) {
#pragma omp parallel for
for (int i = 1; i < n - 1; ++i) {
y[i] = (x[i + 1] - x[i - 1]) / 2;
}
sync(y, n);
}

// The same loop, this time parallelized via FunHPC. Each thread
// handles a well-defined amount of work, to be chosen based on the
// complexity of the loop kernel.
// - Each thread runs until its result is needed. The interior of the
// domain can be calculated overlapping with the synchronization.
// - If the number of iterations is small, only a single thread is
// used. Other cores are free to do other work, e.g. analysis or
// I/O.
// - If the number of iterations is large, then many threads will be
// created. The threads will be executed in some arbitrary order.
// The cost of creating a thread is small (but of not negligible) --
// there is no problem if thousands of threads are created.
void vdiff_funhpc(double *y, const double *x, int n) {
// number of points per thread (depending on architecture and cache size)
// (the number here is much too small; this is just for testing)
const int blocksize = 8;

// loop over blocks, starting one thread for each
std::vector<qthread::future<void>> fs;
for (int i0 = 1; i0 < n - 1; i0 += blocksize) {
fs.push_back(qthread::async(qthread::launch::async, [=]() {

// loop over the work of a single thread
const int imin = i0;
const int imax = std::min(i0 + blocksize, n - 1);
for (int i = imin; i < imax; ++i) {
y[i] = (x[i + 1] - x[i - 1]) / 2;
}
}));
}

// synchronize as soon as the boundary results are available
assert(!fs.empty());
fs[0].wait();
fs[fs.size() - 1].wait();
sync(y, n);

// wait for all threads to finish
for (const auto &f : fs)
f.wait();
}

int funhpc_main(int argc, char **argv) {
const int n = 1000000;
std::vector<double> x(n), y(n);
vdiff(&y[0], &x[0], n);
vdiff_openmp(&y[0], &x[0], n);
vdiff_funhpc(&y[0], &x[0], n);
return 0;
}

===To-Do===

This is a wiki -- everybody should add missing items here

* Maybe: Make FunHPC compile with Clang on Darwin
* Maybe: Set up FunHPC on Bethe or Fermi (if Frank can't get access to Wheeler)
* Add pointers to http://cppreference.com to wiki (for async, future)
* Describe future, shared_future; async's launch:: options
* Make sure all FunHPC examples run on Wheeler
* If possible: look at weird performance numbers (350 ms vs. 3500 ms on Wheeler's head node); run on compute node instead?

====Done:====
* Correct broken FunHPC grid self-test
* Provide make wrapper for Wheeler
* Describe Cereal patch
* Add pointers to package web sites to build instructions
* Put loop parallelization example onto wiki (and make it compile)
* Announce next meeting (Wed Dec. 14, 12:00 EST)

==Mini-Workshop #2: Wed, Dec 14, 2016, 12:00 EST==

(By popular request, later in the day.)

Topic: FunHPC in Cactus

Venue: Google Hangouts https://hangouts.google.com/call/jjkffrrvmnbhrooiyjxhfeb2ume

Agenda:
* FunHPC vs. OpenMP (recap from last week)
* Overview of current proof-of-concept code
* Look at current benchmark results
* How to download, build, run, benchmark
* Next steps

===Source code===
* FunHPC and its dependencies, as described last week (see above)
* FunHPC arrangement, basic thorns and examples (get it at https://bitbucket.org/eschnett/funhpc.cactus.git)
* Cactus flesh, branch eschnett/funhpc (mostly for startup)
* Carpet, branch eschnett/funhpc (mostly to disable OpenMP)
* Kranc, branch eschnett/funhpc (to generate FunHPC-parallelized code)
* McLachlan, branch eschnett/funhpc (contains new FunHPC-parallelized thorns with "_FH" suffix)

I recommend using Wheeler to build and run, as I've tested this.

===To Do===
* Put example parameter file online
* Check whether a vanilla checkout builds
* Update benchmark results
* Next meeting: Next Wed, 12:00 EST (tentatively); continue with FunHPC in Cactus

2016-05-30T09:43:15Z

Eschnett: /* NEEDS-ANALYSIS (results are there, but did not look at them yet) */

====Release Coordination====

There are several places where we coordinate about this release:

* Release_Process
* Release_Details
* Release_coordination (this page)
* TRAC ticket

Make sure you check all places for information.

Once a specific issue has been identified, please create a ticket and move discussion there, and add the release milestone to the ticket. This page is just for coordination of "the test failures".

=Put Stuff Here=

Please list here, with your name, the machines which you care about.

===Activity===

Please describe what you are working on to avoid duplication of effort.

=Testing on various machines=

Erik Schnetter is running the tests on various machines. The current status is:

==NEEDS-RUNNING:==

* stampede-mic
* titan

==RUNNING:==

* mp2
* sunnyvale

==NEEDS-ANALYSIS==

This means that the runs completed, but I did not look at the results yet.

* bluewaters
* orca
* philip
* qb
* smic

==GOOD==

This means that test results are available on https://build.barrywardell.net/view/EinsteinToolkitMulti/job/EinsteinToolkitReport-sandbox .

* bethe
* comet
* cori
* gordon
* gpc
* nvidia
* redshift
* zwicky

==BAD: (something went critically wrong)==

* datura: not accessible (2016-05-30)
* edison: password expired (2016-05-30)
* guillimin: build failure (2016-05-30)
* mike: no access (2016-05-30)
* shelob: cannot log in (2016-05-27); probably no LSU allocation; need to contact frank
* stampede: allocation problems (2016-05-27); xsede portal has allocation; wait and retry