ESMF_MeshCreateDual() to keep the original nodal coordinates as dual element coordinates - and preserve distributions

[theurich]

I am using the ESMF_MeshCreateDual() method to construct the dual mesh (swapping nodes and elements) of the NEPTUNE mesh. The original mesh is constructed with nodal and element coordinates -- as it has to be to support the call into ESMF_MeshCreateDual(). However, I have noticed that the returned dual mesh only keeps the original element coordinates, now as dual nodal coordinates, but drops the original nodal coordinates. As a consequence it is not possible to use fields built on top of the dual mesh with ESMF_MESHLOC_ELEMENT as the source field in bilinear or patch regridding operations.

My request is to keep the original nodal coordinates as the element coordinates of the dual mesh.

A secondary request is to also preserve the distributions such that the dual nodes are distributed as the original elements (I think that is already happening now), and the dual elements are distributed as the original nodes (not happening right now!). It is simple enough to work around the distribution issue by using ESMF_MeshCreate(fromMesh) by swapping nodal and element DistGrid between original and dual mesh, but it would be very convenient to have ESMF_MeshCreateDual() automatically preserve the distributions.

[theurich]

Bob has made progress on branch https://github.com/esmf-org/esmf/tree/fix_meshdual to keep the original node coords as element coords of the dual mesh. This works as expected now on the mentioned branch.

An outstanding issue is with the distribution of the dual elements. Their distribution is currently not guaranteed to be identical to the original mesh nodal distribution.

For instance for a particular setup I am testing with (on a single PET for simplicity) I compare the order the nodal coords (lon/lat) of the original mesh with elements of dual mesh. They are in the same order:

However, comparing the order of elements of original mesh with nodes of the dual mesh shows some swapping in the order:

It is pretty straight forward to work around this issue by accessing the nodal and element DistGrids of the original Mesh, and then using MeshCreate(fromMesh) on the dual Mesh with swapped Distgrids, to get the desired preservation of distribution:

meshDualRedist = ESMF_MeshCreate(meshDual, nodalDistgrid=eDistgrid, elementDistgrid=nDistgrid, rc=rc)

The dual mesh elements are still distributed as the original mesh nodes:

Plus the dual mesh nodes are distributed as the original mesh elements: