Maxwell CarpetRegrid2 Issues
At Erik's suggestion (off-list), I resolved some checkpoint recovery issues by moving from "Curie" to the new ETK release. However, my old grid specifications seem no longer to work.
The parameters are for a 2:1 mass-ratio BHB evolution. I've used three "centres" -- one each for the punctures, and one centred at the origin. Below is what I used to have in CarpetRegrid2, which worked with the last "Curie" release of ETK (the full grid is [-2048,2048] in size).
- Parameters of thorn CarpetRegrid2 (implementing CarpetRegrid2)
CarpetRegrid2::freeze_unaligned_levels = "yes" CarpetRegrid2::num_centres = 3
CarpetRegrid2::num_levels_1 = 12 CarpetRegrid2::position_x_1 = -3.3333333333333333333 CarpetRegrid2::movement_threshold_1 = 0.10 CarpetRegrid2::radius_1[5] = 40 CarpetRegrid2::radius_1[6] = 20 CarpetRegrid2::radius_1[7] = 10 CarpetRegrid2::radius_1[8] = 5 CarpetRegrid2::radius_1[9] = 2.5 CarpetRegrid2::radius_1[10] = 1.25 CarpetRegrid2::radius_1[11] = 0.625
CarpetRegrid2::num_levels_2 = 13 CarpetRegrid2::position_x_2 = 6.6666666666666666667 CarpetRegrid2::movement_threshold_2 = 0.05 CarpetRegrid2::radius_2[5] = 40 CarpetRegrid2::radius_2[6] = 20 CarpetRegrid2::radius_2[7] = 10 CarpetRegrid2::radius_2[8] = 5 CarpetRegrid2::radius_2[9] = 2.5 CarpetRegrid2::radius_2[10] = 1.25 CarpetRegrid2::radius_2[11] = 0.625 CarpetRegrid2::radius_2[12] = 0.3125
CarpetRegrid2::num_levels_3 = 6 CarpetRegrid2::radius_3[1] = 1024 CarpetRegrid2::radius_3[2] = 512 CarpetRegrid2::radius_3[3] = 256 CarpetRegrid2::radius_3[4] = 160 CarpetRegrid2::radius_3[5] = 96 CarpetRegrid2::regrid_every = 60 CarpetRegrid2::verbose = "yes"
This doesn't work now, however. Current ETK complains that some entries in radius_1 and radius_2 are unspecified, and it sets them to zero with catastrophic results.
To resolve this, I am now explicitly setting the missing levels:
CarpetRegrid2::radius_1[1] = 1024 CarpetRegrid2::radius_1[2] = 512 CarpetRegrid2::radius_1[3] = 256 CarpetRegrid2::radius_1[4] = 160
CarpetRegrid2::radius_2[1] = 1024 CarpetRegrid2::radius_2[2] = 512 CarpetRegrid2::radius_2[3] = 256 CarpetRegrid2::radius_2[4] = 160
This runs, but complains every time step about mismatched "simulated domain volume" and "reduction weight sum":
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999999072 WARNING level 1 in thorn CarpetReduce processor 0 host r186i2n12.p4.nas.nasa.gov
(line 120 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetReduce/mask_test.c): -> Simulation domain volume and reduction weight sum differ
After a short evolution time (7.25 M, or more than 1700 time steps), the run crashes. First there are several messages of the form:
INFO (CarpetReduce): Simulation domain volume: 442368 INFO (CarpetReduce): Reduction weight sum: 442367.999991311 INFO (CarpetRegrid2): Centre 1 is at position [-3.29831,-0.331478,7.81462e-15] with 12 levels INFO (CarpetRegrid2): Centre 2 is at position [6.58787,0.867484,8.98107e-15] with 13 levels INFO (CarpetRegrid2): Centre 3 is not active INFO (CarpetRegrid2): Regridding INFO (CarpetRegrid2): Regridding levels 10 and up WARNING level 1 in thorn CarpetLib processor 58 host r186i3n7.p4.nas.nasa.gov
(line 182 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc):
[...]
->
/home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc:948:
[ml=0 rl=10 c=59] The following grid structure consistency check failed: Synchronisation and boundary prolongation: All points must have been received needrecv.empty()
[...] WARNING level 0 in thorn CarpetLib processor 94 host r186i3n12.p4.nas.nasa.gov 12676 (line 1973 of /home1/bjkelly1/CODES/Cactus/configs/hahndol/build/CarpetLib/dh.cc): 12677 -> The grid structure is inconsistent. It is impossible to continue.
I'm attaching my current parameter file in full. I'm reluctant to post the entire STDOUT/STDERR, since it's around a MB in size.
Any ideas would be appreciated. I would turn to the copious documentation on CarpetRegrid2, but as noted before, there is none.
