Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* Todo list for the summer * cleaning the test file definitions * Point gs.bp path * Added a copy to jarvis_utils * Modify tests to copy all needed files to the build and modify the supporting adios documents to operate on local files * Included tests for parallel execution of the gray-scott application. * Changes to the paths on the native tests * Changed test names to make them more descriptive * Sicne we are copying the cleaning script, lets call it at tghe end of every simulation test * Change of paths to enable testing * Added adios_config absolute path * Initial modifications to run the gray-scott test with the hermes engine. Added a new adios2-hermes.xml file * Added test_pdf_calc * Changed path to adios2-hermes.xml when creating engine * Reduced steps and checkpoint freq. Changed adios_config path * Added test_pdf_calc * Added absolute path to adios2.xml * Cleaning code and changes in test names * Added dependencies and test enabling * Change script name: native_tests.py * Load class native_tests * Added class Copy * Added functions for cleaning and preparing simulations * New clean_simulation method using Rm from jarvis-util * Applied 2 processes instead of 4 to avoid memory issues * Enable basic test * Paths to remove in cleanup updated * Removed include hermes-engine.h * Removed absolute path of adios2.xml * Changed steps and checkpoint freq * json for testing Gray-Scott with Hermes * Added tests with Hermes and modified cleanup method * Enable hermes tests * Run the test with just one adios.xml file * Cleaned the code and deleted the executable path * Included the test mode in the prepare_simulation function * Change test names * Reduced the steps performed on the tests to see if the issue on Github Action was related to memory issues, since the tests are running locally * Saving the output of the test for later comparison * Update cmake.yml added output on failure option to the crest to get more information on the issue * Enable "-r" for copying directories * Storing outputs for future comparation * Removed comment * Update cmake.yml Changed ctest option to verbose as output on failure doesn't work on stacked jobs * Copy results after stop_daemon * Change to 100 steps instead of 10 * Change line separator * Save tests outputs in a single results folder * Fix mkdir of results directory * env.sh for future readme * eliminate file * Update Dockerfile with debugging tools * Updated the excluded path on the lint.sh * Changes to the gray-scott so that the docker cna find some of the libraries * Update to the test_manager to fix our issue with the testing and LD LIBRARY PATH * Removed comments * Added and ifdef statement to save the outputs in a text file * Update CurrentStep function * Update CurrentStep function * Fix lint issues * Fix lint issues * Added put_test and get_test * m_CurrentStep private * add basic three vars test * add basic three vars test * Updated engine and tests to work with steps * basic three vars test in two different processes * Removed comments * Removed put test (one process) * Fix lint issues * Fix lint issues * New put and get tests with two variables * change var name from var1 to var * Organize tests in directories * Added spdlog to the project * Refactoring tests locations to make the project more organized * Forgot to add the new CMake files to git. Added them and minor changes to the names on the cmake to adapt to it * Added spdlog. Disable installation with cmake due to conflicts with the header only library. Will make an issue and come back to it on a later data * Aadded missing cmakelist on the spdlog folder * Modified Put method: Storing metadata in bkt and extracting MPI processes information * Allow multiprocess updates * Fixed race conditions issues * Added Cereal * Set upo the cmake file * Changes to the hermes engine to simplify the currentStep call * Changes to the make files for cereal * Implemented the metadata serializer * Added the metadata serializer to the hermes engine * Added a unit test to check the correctness of the serializer * Changes to the metadata struct and the test to easily print the resulting struct * Added cmake for the metadata test * Updated engine to get metadata and use the new serializer * Updated metadata serializer function * Added split metadata test * Modified beginStep for metadata extraction and fixing lint issues * Template for defining and registering variable into the io * Added AddVariable function * Update gitignore * Aded suppor to the metadata serializer to serialize the variable type * Added jarvis utils and jarvis cd as submdoules of the project in preparation for changes that have to happen * Added a remove command to the jarvis utils filesystem class * Added a getDataType for easy retrieval of the type as an adios2::DataType * Perform put/get metadata tests * add jarvis-util * add jarvis-util * add lisOfVars containing variables used * Update BeginStep to retrieve all used variables. * Add submodule: jarvis-cd * Enable all testing * Benchmark testing * Benchmark testing * Update bash to cd build-remote * Clean spaces * Removed RegisterCreatedVariable since we don't need it * Enable defining variable after inquire quey * fixed lint issues * fixed lint issues * fixed lint issues * Added number of process to run in pytest_benchmark * Added number of process to run in test * Added number of process to run in test * New TestManager class for benchmark testing * Remove comments * Commented MACRO for future switch-case replacement * Enable DEFINE_VARIABLE macro * Update to the submodule * changed the linter exclsion paths * Added a test to check the usability of spdlog as a logger * clean up of code with the introduction of HermesPut and Get the get is not fully used through the code Also introudced a mixed spdlog logger for both file and console. TOD: make the console disable itself in the full build and print the function name Also cleaned the code a little bit to better reflect function names and the like * Cleaned the code, changed metadata bucket structure and removed inquire variable from if condition * Commented tests * Prints for log * Enable test benchmarking * Update hostfile * Update hostfile * Change nº of processes * Change nº of processes * Change nº of processes * Change nº of processes * Change nº of processes * Added hostfile path * Added hostfile path * Changed number of process * Changed number of process * Changed path of output file * Changed path of output file * Added logs * Initial approach to MinMax: - VariableMinMax: Override function with the desired functionality - ApplyElementMinMax: Check type and obtain max and min values * Updated hosts * Updated hosts * Changed output * Changed output * Changed output * Changed nº of processes for testing * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Changed nº of processes for testing and file output path * Added minmax logs * Changed parameters for testing * Changed parameters for testing * Initial approach to MinMax: - VariableMinMax: Override function with the desired functionality - ApplyElementMinMax: Check type and obtain max and min values * Changed parameters for testing * Added hostfile path * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed parameters for testing * Changed numAggregators from 1 to 4 * Changed numAggregators back from 4 to 1 and set AsyncWrite to true * Added Substream key * Switch params back * Added AggregatorRatio * Back to first configuration * Updated logs * change output to /mnt/nvme * Change number of processes for testing * Solving Lint errors * Solving Lint errors * Trying to run app in ares * Configure settings-files.json * slurm added to test * Testing with slurm * Testing with slurm * Testing with slurm * Updated mpi_exec_info * Incorporate "test" variable for analysis * Extraction of Execution Time logs * Update CurrentStep Value to start in 0 * New considerations: - Consider the total steps of the writer - Removed comments - Analysis working * Extraction of total number of steps with m_IO.m_Name identification * New version of the Min Max function to consider all types of variables * Updated host issues * Enable execution from compute node with SshExec * Logging execution times * Setting log directory * Removed comments * Removed user and pkey from ssh info * Testing the 4 scenarions from one node * Execute test from the main node * Upgrade number of processes to run * Update all tests to run with slurm * Added orangefs config file * Added orangefs deploy and terminate files * Added test for benchmarking BP5 in NVMe and HDD * Added test for benchmarking BP5 in NVMe and HDD * Enable test for benchmarking BP5 in NVMe and HDD both simulation and analysis * Updated output paths to /client * Added logs for beginstep in analysis (load metadata) * Update engine variable structure and mount Hermes * Just need to find hermes library * Remove temporary host file * making the looger test output line, funcion and file Also, testing the variety of erros that can happen * Updated gitignore * added current step unit test * added current step unit test * Interfaces for Hermes and Bucket designed for unit testing * Implementations of the new interfaces * Changed engine to use the new communication classes for Hermes and Bucket * Updated gitignore * Cleaned test, added GTest and move things around * A bit more: Cleaned test, added GTest and move things around * Added rapidJson * Cmake for rapidJson * Added gtest basic test, Json class and Json tests * Added class loader * Added error codes * Adeed MPI support for Node root detection * Changes to the hermes interface and mock system * Added Threadpool * added aqlite operations * Modified dockerfile to include sqlite * Started the update on the engine to adapt to all the technologies included * Upgrade versions of jarvis utils and jarvis-cd --------- Co-authored-by: jbenegassi <[email protected]> Co-authored-by: Augusto Diaz-Leante Ferreiro <[email protected]> Co-authored-by: Augusto Diaz-Leante Ferreiro <[email protected]>
- Loading branch information
4 people
authored
Sep 7, 2023
1 parent
d0d685b
commit a34bfb8