Add vertical remap feature #4968
Conversation
|
Running list of testing Comparison with E3SM/master as of
Comparison with E3SM/master as of
Compass
|
xylar
added
mpas-ocean
BFB
|
@jonbob and @vanroekel Do either of you have recommendations for the 10-year B-case simulation recommended for stealth features under the new testing procedures? Thanks! |
|
I agree with @vanroekel that the low-res WC v2 configuration is correct for any longer comparison runs |
|
@jonbob My understanding from our slack conversation was that the 50-year G-case simulation I've already performed would be sufficient for this stealth feature. Let me know if your thoughts changed. |
|
@cbegeman - your testing looks great. I'll do more testing before merging, but this is clearly ready and just has work its way through the backlog. |
|
FYI, I intend to add the commit from this branch ocn/add-vert-remap-design-doc, containing the design doc for VLR, after this PR is merged #4973: |
|
@cbegeman thanks for your hard work on this. I am testing now. Do you have results from the idealized tests in Petersen et al. 2015? If not, I can run those and post results as part of my review. In particular, the overflow and internal gravity wave tests provide snapshots and quantitative measures for vertical advection. |
|
@mark-petersen Thanks for testing! Yes, the rpe results from the baroclinic channel and internal wave test cases are: I don't have the overflow test case results. If you have that test case handy in legacy, feel free to give it a whirl. |
There was a problem hiding this comment.
Approved based on my careful review of E3SM-Ocean-Discussion#6, a quick look back through the code, and @cbegeman's testing.
|
@dengwirda, can you take a look at this as soon as possible? |
|
@philipwjones, could you make sure we're not missing OpenACC directives? If we are, suggestions would be welcome! |
|
Testing on badger. Gnu debug fails for RK4 and the threads test, when I add the results are: With the default |
|
@mark-petersen Thanks for alerting me to this. The RK4 tests are expected to fail (we decided not to support RK4) but the threads test is a surprise to me. I'll look into it. |
|
@mark-petersen Hmm, I'm having trouble replicating your error. I ran Any idea why? Maybe @xylar has thoughts? |
|
@cbegeman, I'm able to reproduce the error. I'm seeing: @cbegeman, could it be that you're running with an older compass that doesn't see |
|
My results are in: in case they're helpful |
|
@cbegeman, thanks for the fix in 67ca053. I must say I don't understand why that fixed the threading error but glad it did.
I don't know how timers relate to threading at all. @mark-petersen, do you have any idea why some timers show up and others don't with treading enabled |
- Add config option to turn remap on/off - Add config option for advection, remap order - Add config option for target grid - Only call vert_transport_top if advection_method is flux-form - Construct opts for remapping using config options
- vertAleTransportTop = 0 so no vertical velocity during momentum solve - Add computation of vertVelocity from lagrangian layerThickness to diagnostics routines
- Add option for WENO slope limiter - Fixup config name for vertical tracer adv order - Add option for remapping every interval number of timesteps - Update namelist entries - Add checks on config options
cbegeman
force-pushed
the
ocn/add-vertical-remap-feature
branch
from
6e34a3d to
e30698f
Compare
|
test merge passes:
|
…4968) Add vertical remap feature This PR adds the option for vertical Lagrangian remapping (VLR). The idea behind VLR is to allow the vertical interfaces between layers to evolve in a Lagrangian fashion, bypassing the Eulerian vertical advection scheme and removing the vertical CFL limit. The layer interfaces are then remapped to a target grid every interval of timesteps. This development includes the following features: * A new vertical remapping module mpas_ocn_vertical_remap.F * Adds PPR library as a submodule of the ocean component used to perform conservative remapping * Config option to turn remapping on/off: on = config_vert_advection_method = 'remap' off = config_vert_advection_method='flux-form' (default) * Config options for remapping: the order of remapping, the remapping interval and the remapping limiter * A diagnostic output file containing remapping-related variables * Computation of the Eulerian vertical velocity, which is nonzero at timesteps at which regridding and remapping occur * Vertical tracer advection budget terms which must be computed after this Eulerian vertical velocity is computed. Thus, for the remapping case this computation is performed (as usual) in ocn_tracer_advection_mono_tend, where the vertical terms are zero, and repeated in diagnostics after remapping, where these terms may be nonzero. * Timer for remapping named "vertical remapi". Roughly 20% of runtime is spent remapping when remapping occurs every timestep. However, in most realistic cases remapping at intervals of 3-5 timesteps are acceptable, reducing performance decreases to ~5%. VLR is currently only supported for split-explicit time integrator. Regridding is hard-coded as z-star. Support for alternate grids is planned. [NML] [BFB]
|
also passes:
merged to next |
|
merged to master and expected NML DIFFs blessed |
|
Yay! Congratulations, @cbegeman, this was a ton of work on your part. |
|
Thanks @xylar, @jonbob, @dengwirda, @mark-petersen and @philipwjones for working with me on this! I'm happy to see this in E3SM! |
This PR adds the option for vertical Lagrangian remapping (VLR). The idea behind VLR is to allow the vertical interfaces between layers to evolve in a Lagrangian fashion, bypassing the Eulerian vertical advection scheme and removing the vertical CFL limit. The layer interfaces are then remapped to a target grid every interval of timesteps. Griffies et al. (2020) provides an overview of VLR.
This development includes the following features:
mpas_ocn_vertical_remap.Fconfig_vert_advection_method = 'remap', off =config_vert_advection_method='flux-form'(default)ocn_tracer_advection_mono_tend, where the vertical terms are zero, and repeated in diagnostics after remapping, where these terms may be nonzero.vertical remap. Roughly 20% of runtime is spent remapping when remapping occurs every timestep. However, in most realistic cases remapping at intervals of 3-5 timesteps are acceptable, reducing performance decreases to ~5%.Further discussion and simulation results can be found at E3SM-Ocean-Discussion#6
[NML]
[BFB]