FullyAssembledRows vs SubAssembledRows for GMG

[JordiManyer]

One of the performance inefficiencies we have right now (both in GMG and patch-based smoothers) is the fact that we are constantly having to copy information between PVectors with a matrix-ghost layout (used in CG) and a FESpace-ghost layout (used in the transfer operators).

Within a conversation with Santi, he thought that restricting our implementation to FullyAssembledRows assembling strategies could resolve this issue and would result in greater efficiency.

Food for thought...

[amartinhuertas]

One of the performance inefficiencies we have right now (both in GMG and patch-based smoothers) is the fact that we are constantly having to copy information between PVectors with a matrix-ghost layout (used in CG) and a FESpace-ghost layout (used in the transfer operators).

Can you please point in the code the particular point at which this copy is being done?

[JordiManyer]

For instance:

• In transfer operators for GMG here, here, here, here and many other places.
• I patch based solvers here

It's an issue we've discussed a couple of times already...

[amartinhuertas]

It's an issue we've discussed a couple of times already...

I am aware of this.

It is just I don't have the code in mind, so that it is quicker if you point me to the particular lines of code rather than myself trying to find them.

Thanks.

[amartinhuertas]

Within a conversation with Santi, he thought that restricting our implementation to FullyAssembledRows assembling strategies could resolve this issue and would result in greater efficiency.

I am not sure how FullyAssembledRows is going to fix this.

If am not wrong, the issue is that the local numbering of DoFs in the FESpace does not match the local numbering of DoFs in the linear algebra data structures. Perhaps I am missing something ...

[amartinhuertas]

BTW ... did we measure the overhead of these copies? It is actually an issue/hot spot in practice?