Dynamic topology region file opening with group I/O #469
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Basic idea is to create a topology modification observer that can be registered on the region. This provides a callback mechanism for re-defining the Region mesh when a topology change occurs. The API is still currently fluid and will change until something is settled upon. A pre-requsite to this was adding the capability for the Region to handle file creation such that it can either create multiple output files when the topology changes or uses the NETCDF capability to create multiple file groups within a single file. Unit tests are included to show how use cases drive the development
…n to the Catalyst IOSS Database (sandialabs#456) * Refactored testing infrastructure to add transient unstructured fields * First test failing, for wrong reason * Test Fixture completed. First failing test done. * Added failing tests for all 5 Props. Need to update Frwrk for Sur_Split_type * Refactored get*block() to be public * Added failing list for SURFACE_SPLIT_TYPE. Compromises made * Fixed conduit error related to scoping. Refactored tests a bit * Implemented Exodus IOSS properties. Removed asserts. Finished verifying fields * Refactored readField() to not have DatabaseIO as parameter
* EXODUS: refactor field metadata attribute output * EXODUS: Beginnings of basis read * EXODUS: basis query * EXODUS: these got lost with some branch mishaps * EXODUS: Spelling fix * EXODUS: See if can fix intel errors * EXODUS: Fix intel build * EXODUS: Move internal function to correct header * Windows fixes: make functions non-inline Making certain functions in Ioss::Utils non-inline to avoid export errors on Windows. Accessing the non-exported static variables was causing errors. This avoids that. Also added API to access current values for various streams. * eliminate warnings * IOSS: cgns - do not create assemblies from Unspecified VC * Getting kokkos build working again [ci skip] * TEST: Enable kokkos build * IOSS: Change byte-size output to debug * shellcheck recommendations * TESTING: Remove adios2 until figure out issues * EPU: Minor improvement in message (plural vs singular) * IOSS: Rename qsort to sort * IO_MODIFY: Better help output * IO_MODIFY: clean-up comment [ci skip] * Revert "EPU: Minor improvement in message (plural vs singular)" This reverts commit 6ca490c. * IOSS: Switch to using doctest from Catch2 * IOSS: cgns - remove lambdas to see if fixes stria issue * IOSS: Remove shadowed variable * Revert "Remove shadowed variable." This reverts commit 4d54381a3264dea155de65d0588cb192b37c0e1f. * Revert "IOSS: cgns - remove lambdas to see if fixes stria issue" This reverts commit 8dde87a74344b8b57f0f948e58f73e82c950e074. * IOSS: cgns - get CG_BUILD_PARALLEL define for all versions * Update readme -- replace Catch2 with doctest [ci skip] * @wortiz has signed the CLA from Pull Request sandialabs#303 * @mwestphal has signed the CLA from Pull Request sandialabs#304 * BLOT: Fix new argument mismatch warning with gcc-12 * Fix a recursive warning on assignement copy operator * IOSS: DEBUG->DbgOut, WARNING->WarnOut to avoid windows issues * IOSS: new version of clang-format * New version of clang-format * New version of clang-format * TPL: Add INSTALL_PATH to metis/parmetis installs * MODULE: Remove old unused unsupported lines [ci skip] * @gjtempl has signed the CLA from Pull Request sandialabs#305 * Updates for serial build Add SEACAS_HAVE_MPI include guards for ioss unit_tests that use MPI. These are still run for serial builds and this change allows them to compile and run. Add -DMATIO_SHARED:BOOL=${SHARED} to TPS/matio/runcmake.sh. Matio's CMake isn't respecting the CMAKE variable BUILD_SHARED_LIBS and still builds shared version when SHARED=NO. * IOSS: Handle uppercase field suffices * IOSS: Clean up uppercase suffix fix * ZELLIJ: Add offset option * KOKKOS: Use current version * TPL: Fix kokkos runcmake.sh for current version * EXODUS: Clean up * Copy top-level README to packages/seacas so available in Trilinos snapshot * CONFIG: Update name of kokkos library * EXODUS.PY: Add the get_block_id_map method This adds functionality to add being able to easily find the elements contained within EX_ELEM_BLOCK or other type without keeping track of global and internal indicies. * ZELLIJ: Better cache utilization for offset and scale * IOSS: Handle specified field ordering (via index) if present * IOSS: use reserve instead of resize since pushing back * IO_MODIFY: Add rename capability * IO_MODIFY: Clean up rename capability * CONFIG: Add gnubrew compiler option * CONFIG: Update hdf5 versions * IO_MODIFY: update help options * EXODUS: Fix problem with ex_get_block_id_map function * SUPES: Make a variable be a correct spelling * EXODUS: clang-analyzer fixes * Bring up-to-date with master * Clean up bad merge * EXODUS: Clean up bad merge * EXODUS: Clean up anothr bad merge * EPU: Bring in fix from master * EXODUS: Clean up python and varid lookup * EXODUS: Support cardinality and user-define suffices; refactor * EXODUS: Further refinement/refactoring of DG field interface * EXODUS: Refactor component separator to allow for empty value * EXODUS: Minor cleanup * EXODUS: Refactor function naems; add test of ex_field utils * EXODUS: Add missing file from last commit * EXODUS: Fix bad handling of user-defined suffices * EXODUS: Fix suffix handling in name building * CI: Fix cmake version requirement * EXODUS: More work on the basis metadata read * EXODUS: Fix null test -- use double pipe * EXODUS: refactor field metadata attribute output * EXODUS: Beginnings of basis read * EXODUS: basis query * EXODUS: these got lost with some branch mishaps * EXODUS: Spelling fix * EXODUS: See if can fix intel errors * EXODUS: Fix intel build * EXODUS: Move internal function to correct header * TEST: Enable kokkos build * EPU: Minor improvement in message (plural vs singular) * Revert "EPU: Minor improvement in message (plural vs singular)" This reverts commit 6ca490c. * IOSS: Remove shadowed variable * Revert "Remove shadowed variable." This reverts commit 4d54381a3264dea155de65d0588cb192b37c0e1f. * CONFIG: Add gnubrew compiler option * Bring up-to-date with master * EXODUS: Support cardinality and user-define suffices; refactor * EXODUS: Further refinement/refactoring of DG field interface * EXODUS: Refactor component separator to allow for empty value * EXODUS: Minor cleanup * EXODUS: Refactor function naems; add test of ex_field utils * EXODUS: Add missing file from last commit * EXODUS: Fix bad handling of user-defined suffices * EXODUS: Fix suffix handling in name building * CI: Fix cmake version requirement * EXODUS: More work on the basis metadata read * EXODUS: Fix null test -- use double pipe * EXODUS: Test field metadata read/write on nodes * CI: Fix cxx standard setting * EXODUS: Fix bad merge * EXODUS: Fix bad merge * EXODUS: Fix varid query for assembly and blob * EXODUS.PY: See if this fixes ci build issues * EXODUS: Fix ex__get_varid function * Revert "EXODUS.PY: See if this fixes ci build issues" This reverts commit 86f202d. * EXODUS: Clean up ex_attribute struct * EXODUS: realpath is gnu extension in c11/c99 * EXODUS: Provide realpath prototype * EXODUS: Add missing functions for mingw * EXODUS: Fix ex_attribute initiialization * CI: Disable nczarr in netcdf * EXODUS: Fix codacy null / 0 issue * EXODUS: Removed unused struct member * IOSS: Clean up some includes; remove the ... for ??? * EXODUS: Fix bad master merge * EXODUS: Fix intel compiler build * BLOT: Increase resolution of node/element id output * IOSS: Change meta_data to metadata * EXODUS: Use internal naming for static function * Remove ioss as required library * EXODUS: Add function name to output so know where it came from * IOSS: Add function mapping Ioss field names to exodus ex_field_type * IOSS: Output field metadata basic implemetation * IOSS: Rename function for consistency * IOSS: Add missing serialize io call * APREPRO: Make strings test more robust * IOSS: Build io_modify only if exodus enabled * IOSS: Detect if exodus enabled for exodus include * GREPOS: Intel overoptimizes unless inimap separate file * GREPOS: Fix bad merge * CONFIG: Silence CMake warning about python interpreter finding * IOSS: Fix use of c++14/17 type-traits * EXODUS: Filter out attributes Field@ * IOSS: Add mapping from exodus type to ioss field type * IOSS: Handle composite fields with different component separators * IOSS: Field - Add print method; support 2 separators for composite fields * IOSS: get_fields works after enhanced field found some fields * IOSS: Read/interpret exodus files with field metadata * IOSS: Eliminate some char **names uses -- convert to NameList * IOSS: Use more NameList and IntVector * CI: Remove cmake policy; not recognized in docker build * IOSS: CGNS - earlier versions have the CG_BUILD_HDF5 define in cgnsconfig.h * CI: Update to latest version [ci skip] * IOSS: Minor change to trigger workflows * IOSS: Propogate name_array changes to parallel * IOSS: Add serialize io calls wehre needed * IOSS: Use correct export macro * IOSS: Scalar fields do not need metadata; clutters file also * Io modify handle cgns (sandialabs#453) * IOSS: io_modify - try to get cgns structured mesh coordinate mods working * IOSS: io_modify - fix coordinate modification routines * IOSS: io_modify - try to handle block geometry modify in cgns * IOSS: io_info -- bbox works for structured mesh * IOSS: io_modify - redo geometry transformation code --------- Co-authored-by: Greg Sjaardema <[email protected]> * EXODUS: Fix element count in test writer * IOSS: Handle user-defined field metadata output * IOSS: Handle composite field multiple separators * CI: See if can fix appveyor build [ci skip] * IOSS: Put debug output in ifdef * CI: Actually trigger build... * CI: See if this affects appveyor... * CI: Change install name/location * EXODUS: Fix formatting of comment; turn off clang-format * IOSS: Initial steps in adding Basis capability * IOSS: Initial steps in adding Basis capability to field metadata query * CI: mkdir lib before plugin path in case doesnot exist [ci skip] * APREPRO: Fix some issues found in sierra input files * Testing an undefined variable against a string in a boolean (EQ, NE, LT, ...) was always returning true. . Changed such that an undefined variable in a boolean test with a string expression treats the undefined variable as the empty string. `{if (undefined_var == "Greg")}' Will parse as `{if ("" == "Greg")}' and output an undefined_variable warning. * The `exp()` function will output an ERANGE error for both underflow and overflow. We are ok with underflow not giving an error. Changed so only get math error in case of overflow. * EXPLORE: Add/Refactor qa, qainfo and info list commands * CI: Better error handling in script * EPU: Delete input files if requested * APREPRO: Fix to_string when fmt:: full precision output being used * APREPRO: Refactor last change * IOSS: heartbeat - add HEARTBEAT_FLUSH_INTERVAL property, document 0 will flush every step * EXODUS: Fix output of Curl gradient field * IOSS: Get field-metadata basis somewhat working * APREPRO: Bring branch up-to-date and fix bad merge * IOSS: Refactor function names for ComposedVariable * EXODUS: Move basis to global attribute * IOSS: Add quadrature storage type; beginnings of allowing multiple basis types * EXODUS: Redo so basis, quad stored at global level * IOSS: Modify tokenize to optionally return empty tokens * IOSS: Start of refactor for allowing multiple basis; basis at root of exodus * EXODUS: Allow more than one basis; field can specify which basis it is using * IOSS: Refactor basis struct/type * Iniitial support for quadrature storage type * EXODUS: Treat Quad@ attribute as internal * EXODUS: Add quadrature type and multiple basis/quad types to test read/write * IOSS: cleanup quadrature implementation * IOSS: Better const correctness * IOSS: VariableType -- function to return quad/basis types * IOSS: Support quadrature type in ioss<->exodus type mapping * IOSS: exodus - output basis/quadrature types * IOSS: Better suffix upper/lower detection * EXODUS: Support nesting of user-defiend fields * IOSS: Better compare output for fields * IOSS: Remove debug print * IOSS: Allow nesting on user-defined fields; refactor basis/quad output * EXODUS: Remove unused variable * EXODUS: Output from test program to stdout, not stderr * EXODUS: Add test for enhanced field metadata * EXODUS: Fix pipestatus values * EXODUS: Fix windows build * EXODUS: Remove shadowed variable * EXODUS: Fix enhanced fields for random attribute ordering * EXODUS: Test - more robust when nc4 changes attribute ordering * EXODUS: get_X_metadata functions have count argument added so call-to-call order works * EXODUS: Check for a null pointer * EXODUS: Provide internal strsep and strlcat * EXODUS: Improve portability * EXODUS: A better way to get strdup * EXODUS: Fix some compiler warnings; see if that makes test run... * EXODUS: Fix memory leak in test code * EXODUS: Deallocat struct in correct place * EXODUS: Reorder unit test and update output * IOSS: Add type and type_string to variable types * IOSS: Refactor detailed field output * IOSS: Allow detailed field output * IOSS: Minor cleanup/refactor * NEM_SLICE: Minor fix * IOSS: Clean up fmt includes/use * IOSS: Rewrite how catylyst deals with field names * IOSS: Fix up some changes for catalyst for this branch * IOSS: Fix default separator setting; do not recognize fields if property set * EXODUS: Refactor get/put basis and quad metadata * IOSS: Minor tweak to field printing * IOSS: Refactor variable type determination * EXODUS: Update nesting define; fix test dump file * IOSS: Simplify datapool - remove seldom used capabilty * IOSS: Refactor some names; add print * EXODUS: Eliminate memory leak * EXODUS: Fix spelling error * EXODUS: Fix compiler warning * EXODUS: Clean up some function name/doc * Spelling fixes [ci skip] * IOSS: Add backward compatible interfaces back --------- Co-authored-by: Utkarsh Ayachit <[email protected]> Co-authored-by: Greg Sjaardema <[email protected]> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Mathieu Westphal <[email protected]> Co-authored-by: Gary Templet <[email protected]> Co-authored-by: Mario LoPrinzi <[email protected]>
…ation (sandialabs#458) * Made conduit mimic exodus prop changes in Cat Database on read * Added field_data equality checks btwn Exodus and Catalyst DBs in exo prop tests
* IOSS: refactor to reduce strings; give flexibility and efficiency * IOSS: Start of support for parallel line decomp * IOSS: Explicit template instantiation fix * IOSS: Allow passing filename down into line_decomp * IOSS: initial line_decompose(); compile/link does not run * IOSS: Refactor to try to eliminate duplicate code * IOSS: Add some missing includes * SLICE: Fix data_storage type * Share decomp code between slice and ioss line decomp * IOSS: Fix serial build * Pull element centroid into common utils class * Templative line_decompose; pass correct vector * Handle specified in guided_decompose * Remove unused function * elementToProc does not need 64-bit range * EXPLORE: Fix behavior after bad parse warning * EXPLORE: Better warning/info message on SELECT * Minor rearrange include files * IOSS: See if this fixes/affects msys2 build * IOSS: Add some logging/hwm code to line decomp * IOSS: Another try to see how affects msys2 build * IOSS: Better hwm logging output * Unify Slice and DecompositionUtils zoltan_decompose * IOSS: Fix msys2 build * clang-format run * SLICE: Version should be updated for latest chagnes * IOSS: Reduce storage potentially; fix zoltan free call * SLICE: Fix order of file close and mpi_finalize * IOSS: Enable decomposition statistics for line decomp * IOSS: io_shell - add_processor_id_field works for exodus also * IOSS: compose output will add a proc_id map to output * IOSS: thread-safe output_processor_id_map * CI: safer variable naming --------- Co-authored-by: Greg Sjaardema <[email protected]>
* Correct Jamfile to match Seacas's CMake Library Names Jamfile change for Sierra's build system to support a spack built version of Seacas. This change correctly matches Seacas library names to CMake's version of Seacas's libraries and changes the path name to 'seacas' rather than 'sierra_seacas' Signed-off-by: akimler <[email protected]> * Update Jamfile Fixed accidently removed Jamfile changes Signed-off-by: akimler <[email protected]> * Update Jamfile Remove whitespace Signed-off-by: akimler <[email protected]> --------- Signed-off-by: akimler <[email protected]>
gsjaardema
approved these changes
Jul 17, 2024
There was a problem hiding this comment.
I think that overall it looks very good. There are a few nitpicky comments and or suggestions.
It would be nice to have a short document/comments describing how an application would use this with a few different use cases, but that is probably planned after the initial code is in place.
A few followon suggestions:
- documentation
- modify io_shell to be able to take a set of -s000X files and create the singel grouped file.
- modify io_shell to do the reverse of above -- take a grouped file with topology changes and output the -s000X files.
- (The above two make it easier for the users to try it out)
This may alrady be in the code, but it looks like the DynamicTopology will write either -s000X sets of files, or a singel grouped file. Does it also manage both kinds on read? That is, if a user opens up a database and that database has groups in it, then they can discover the groups and what steps corresons to which groups, get min/max time, step counts, ...
Does the same hold for the case where they open up a database that has the base and then one or more -s000X files -- can they manage that as a single database, or do they have to manually query each one?
Again, looks good and I think there will be lots of requests for enhancements from teh applications/users.
| // number. Assume maximum of 9999 steps (will do more, but won't have | ||
| // good lineup of step numbers. | ||
| // Check database for validity (filename and a type) | ||
| if(m_ioDB == "" || m_dbType == "") |
There was a problem hiding this comment.
if (m_ioDB.empty() || m_dbType.empty())
| if(m_ioDB == "" || m_dbType == "") | ||
| { | ||
| std::string error_message; | ||
| if(m_dbType == "") |
| if(m_dbType == "") | ||
| error_message += "The database TYPE has not been defined\n"; | ||
|
|
||
| if(m_ioDB == "") |
| fmt::print(errmsg, error_message); | ||
| IOSS_ERROR(errmsg); | ||
| } | ||
| assert(m_ioDB != ""); |
Yes, the idea is to write a few doc tests in STK that show how to interface with this. Sierra Fmwk handled all the logic of creating and managing the multi-file case and that was what was moved over into IOSS. However, since STK does not subscribe to Fmwk, the aim is to have it rely on this new behavior in IOSS to replicate the previous Fmwk behavior.
That sounds like an excellent suggestion .. I want to continue working with you to get the API settled down first before implementing this.
This is precisely what one of the unit tests demonstrates
For the multi-file case, I would have to again rely on the previous workflow that exists in Fmwk to support operations that span all the -s000X files but that is definitely do-able.
Cool .. phew :D |
|
Looks good. A few things maybe left, but those can be added as you like... |
gsjaardema
added a commit
that referenced
this pull request
Jul 18, 2024
This reverts commit ace7af0.
gsjaardema
added a commit
that referenced
this pull request
Jul 18, 2024
gsjaardema
added a commit
that referenced
this pull request
Jul 18, 2024
…#472) * Revert "Revert "Dynamic topology region file opening with group I/O (#469)" (…" This reverts commit d828356. * Revert "Revert "Dynamic topology region file opening with group I/O (#469)" (…" This reverts commit d828356. * IOSS: adios2 -- add base_filename to properties not to compare * Another property to ignore Signed-off-by: Greg Sjaardema <[email protected]> --------- Signed-off-by: Greg Sjaardema <[email protected]> Co-authored-by: tokusanya <[email protected]> Co-authored-by: Greg Sjaardema <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial workflow design to support dynamic topology through Region file handling and the use of NETCDF file groups. The file handling is an automatic mechanism that can support the multi-file style output of Sierra Fmwk or single file concatenation using groups.
A topology modification observer that can be registered on the region provides a callback mechanism for
setting up the Region mesh when a topology change occurs.
Unit tests are included to show how use cases drive the development