Adaptivity + Stateful Materials + Repartitioning

[permcody]

Right now we can do any 2 of 3 of these things. We need to be able to marshal and communicate our material properties when the mesh is repartitioned. Our current idea is to rekey our material property database on libMesh's "unique_id" instead of elem pointers.

Note: Material data types are templated so users can use any type they want. dataLoad() and dataStore() must be implemented for every type used as a Material property.

[permcody]

Found a test that is now tripping an assert 100% of the time when running on certain numbers of processors.

stateful_adaptivity_test.i fails with n_cpus = 3:
Assertion `props().contains(child_elem)' failed
Child element pointer is not in the MaterialProps data structure
at /Users/permcj/projects/moose/framework/src/materials/MaterialPropertyStorage.C, line 224

This assertion was only recently added as a sanity check here:
#4641

[friedmud]

Just to be clear: this is failing in the PR... right? Or do you mean that it is failing all the time (like with the current HEAD)?

[permcody]

This is only failing with the PR. I thought it was head but I was mistaken.

[rwcarlsen]

@roystgnr - where would I need to be poking around in libMesh to see how to transfer stateful material properties between processes upon repartitioning? Is there a pattern that you could point me toward that I could follow - e.g. like how elements are sent between processes?

[roystgnr]

Vector data is transferred as part of System::project_vector(), a much more complex operation that probably isn't going to be analogous to anything you want to do. Nodes and elements are copied in MeshCommunication::redistribute() and then (if now unnecessary on their origin processor) deleted in MeshCommunication::delete_remote_elements(). That code is probably both a half-decent example to follow and a good place to hook in to the system. If you add a ghosting functor to the mesh which overloads GhostingFunctor::redistribute() and GhostingFunctor::delete_remote_elements() then it'll get called right where you need it to be. I think mesh_communication.C is decently commented code right now, but if you disagree don't hesitate to send me questions. For the copy operation, we're using send_packed_range() etc. methods, which are necessary if your data is variable-length or overkill if not. To get up and running with those you could look at the unit test in tests/parallel/packed_range_test.C or at the Node and Elem implementations in src/parallel/parallel_{node,elem}.C. @permcody helped me make that framework more useful and is pretty familiar with it too now if you need somebody nearby to ask when I'm not online.

[friedmud]

We currently index stateful data based on Elem pointers... that needs to change to indexing based on Elem::unique_id.

All of the stateful data is already serializable and can be sent as std::string. @permcody already knows how to do that and can show you.

After the adaptivity system has done its thing you can go over the local active elements and look for any that have missing stateful data. There'll then need to be a broadcast where all procs ask for stateful data they are missing. Everyone can receive the lists and then look through their own stateful data (based on unique_id) and find data that needs to be sent out. They can then serialize it and ship it.