non-lockstep coupling of two codes with SST, how?

[liangwang0734]

I have implemented a lockstep demo using SST engine. Doing non-lockstep coupling seems doable but I'd like to learn if you have experience/suggestions.

Background: A uses small time steps, B uses large time steps; Both A and B are parallel programs though I'm testing with 1 processor each.
Goal: A and B exchanges data every 1 second (in physical time not wall time).

My first though was to manually code up some nontrivial logic in the time loops to achieve the coupling. But I think some of the sst options, such as QueueLimit, QueueFullPolicy , etc. are designed to achieve such goals? I'm still learning these options as a end user.

Any thoughts?

[anagainaru]

The RendezvousReaderCount option is by default 1 which means the writer waits until a reader received the step before proceeding. If you set this parameter to 0, your A writer will not block and keep producing steps. However, B will only receive the steps that are available when it finished processing its current step. For example, if A is twice as fast, B will receive only every other step. You can increase how many steps are buffered with QueueLimit.

More about each parameter here: https://adios2.readthedocs.io/en/v2.10.2/engines/engines.html#sst-sustainable-staging-transport

[eisenhauer]

Hi Liang,
Just wanted to chime in on a couple of things. First (in case you were trying it), you should not be doing an open/close for each timestep. Connecting a reader to a writer is indeed a synchronous event, particularly with the RendezvousReaderCount controlling it. But even for later readers, they don't move past Open until the Writer has noticed that they are waiting and they've done a relatively complex handshake. Second, Ana's right that QueueLimit might be what you want. But it kind of depends. The easiest thing to do if you want a timestep exchange every second would be to make sure that A only produces a timestep (does BeginStep/EndStep) every second, which I assume would be after some larger number of your application's steps. You could do that by just checking a timer after your app's steps. (With the caveat that every MPI rank has to make the same decision as to do the Begin/EndStep on this application step, and processor clocks aren't always perfectly in sync, so you might have to do this decision on one rank and broadcast it. A second approach would be to use QueueLimit, produce a step more frequently on the writer, but only consume it once a second on the reader. Steps are essentially queued on the writer until requested (more nuance there), so newer steps will be dropped by the writer if one is already queued. The downsides to this approach is that the writer still experiences costs per Begin/EndStep in the form of data queueing and sending metadata to the reader. Also there is what you might call a "lag". Above I say that new steps are dropped. To avoid a distributed consistency protocol, SST can't discard the oldest timestep when there is a QueueLimit, so it always discards the just-produced one. This often works out to be the same thing, but on human time scales it might be noticable that what the reader gets this time is not what was just produced, but instead the first step that was queued after he finished consuming the previous step.