For multi-tile I/O, allow concurrent reads/writes of the different tiles

[billsacks]

There are performance issues with multi-tile I/O since the tiles are written sequentially (see recent notes in https://github.com/esmf-org/esmf-support/issues/489).

We'd like to investigate being able to write tiles concurrently - either by having multiple PIO instances or by changing something else about the PIO setup to allow concurrency or at least get similar performance to what we'd get from this concurrency.

See notes on this in the 2025-01-15 Core Team Meeting Notes.

[anntsay]

from Core team;

Option 1: if the right way, we need to rework how PIO is initialized in ESMF entirely. Feel like this is the right thing is to initiate once and cleaner. But effort is huge
Option 2: reinitialize PIO as many times as we want. But it will make the rework in the future harder. It would be faster development but still tricky. Maybe a week work? But will need to plan various gotchas.

Bill’s inclination is to go with the quicker and dirtier approach of Option 2: keeping the PIO initialization happening where it currently is, but creating a separate PIO instance for each tile.
Bill: something that might make this hard is: for concurrency, it might be necessary for PIO instance 1 to use a subset of PEs that appear on tile 1, instance 2 to use a subset of PEs that appear on tile 2, etc. Bill isn’t sure if this would be needed for concurrency

Action:
Talk to UFS team on the urgency timeline -see if we could sync this up with 8.9 release in 3 months or so.
Figure out who may be able to take on this
(Bill / Jim / Ann ) Ask Jim E. to see if we could do this at PIO level where it can internally create multiple instances rather than ESMF level. Or could PIO have some other mechanism for concurrently writing these different tiles even given a single PIO instance?