Synchronizing CRS format arrays

[smelchio]

If the number of degree of freedoms is not the same for all tets, we can use a2ab and ab2b to store it. Can the data stored in ab2b be synchronized? For Read array we can use sync_array to do so. But what about varying size arrays?

[ibaned]

You'd need to form a different communication pattern (underlying Dist object). It can be done, but is more expensive and isn't something you can do with existing sync* APIs

[smelchio]

Do you mean that it is more expensive than synchronizing an "equivalent" array with a fixed number of degrees of freedom? Would you still recommend this approach in order to store arrays with a varying size? Or do you think that it would be more efficient to have a fixed size array with the maximum number such that the values are 0 where we would like to store nothing (assuming that we can afford in term of memory and that the maximum number of degrees of freedom is not more than twice the minimum for instance)?

[ibaned]

In general, the two-array CRS approach is the most efficient general way to store varying size data. If the variance is less than 2X, using a fixed size could be cheaper, especially for communication, but only profiling can confirm that.

[smelchio]

Instead of one arrays with a varying number of degrees of freedom, it might be equivalent to work with several arrays such that for each them we store data on a different subset of the mesh (each being identified, e.g., using a Mesh::class_sets -> cf. #307). Would such an approach simplify the implementation? Would the performance be good in terms of communication?

[tristan0x]

CC @WeiliangChenOIST

[smelchio]

This task is now on the top of our stack and we would like to move forward on it. We already implemented the following

struct VariableSizeData {
  protected:
    osh::LOs a2ab;
    osh::Write<T> ab2c; // mapping from (a, b) to the value c of the data of the type specified by b at entity a
    osh::LO ab(osh::LO entity_index, osh::LO data_index) const{
        const auto shift = a2ab[entity_index]; 
        return shift + data_index;
    }
};

but when it comes to synchronize this data structure we of course cannot call the function sync_array in mesh.cppthat relies on https://github.com/SNLComputation/omega_h/blob/master/src/Omega_h_dist.cpp#L108-L122. Could you provide us some support to develop this missing piece?