Merge branch 'improvement.mpi3.restart' into 'master.dev'

[improvement.mpi3.restart] Communicate PartInt,PartData with MPI upon LB instead of HDF5 readin See merge request piclas/piclas!670
piclas-framework · Aug 4, 2022 · 4a0d1bc · 4a0d1bc
2 parents 853c129 + 57f24b5
commit 4a0d1bc
Show file tree

Hide file tree

Showing 59 changed files with 4,205 additions and 2,427 deletions.
diff --git a/REGGIE.md b/REGGIE.md
@@ -8,16 +8,16 @@ PICLas utilizes the Reggie2.0 toolbox for regression testing. A detailed documen
 
 Overview of the test cases performed after a commit.
 
-| **No.** |                  **Case**                   |                  **CMAKE-CONFIG**                  |                                                                                   **Feature**                                                                                    | **Execution**  |   **Comparing**   |                       **Readme**                       |
-| :-----: | :-----------------------------------------: | :------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------: | :---------------: | :----------------------------------------------------: |
-|    1    |                  run_basic                  |                    maxwell,RK4                     |                                                                                   DG-Operator                                                                                    | nProcs=1,2,5,8 |      L2,Linf      |                                                        |
-|    2    |                 CHE_maxwell                 | maxwell, RK4, Particles=OFF, PICLAS_DEBUG_MEMORY=T |                                                            DG-Operator, only Maxwell field solver, hard compiled N=1                                                             | nProcs=1,2,5,8 |      L2,Linf      |                                                        |
-|    3    |             CHE_poisson/poisson             |                    Poisson,RK3                     | HDG-Operator, secondary electron emission (SEE-I model, does not happen because ions are too slow), hard compiled N=1, CalcBoundaryParticleOutput=T,CalcElectricTimeDerivative=T | nProcs=1,2,5,8 |      L2,Linf      | [Link](/regressioncheck/CHE_poisson/poisson/readme.md) |
-|    4    | [CHE_PIC_maxwell_RK4](#che_pic_maxwell_rk4) |                 PIC (maxwell, RK4)                 |                                                                            PIC-variableExternalField                                                                             |                |                   |                                                        |
-|    5    |            [CHE_DSMC](#che_dsmc)            |                        DSMC                        |                                                                                                                                                                                  |                |                   |                                                        |
-|    6    |         CHE_PIC_maxwell_implicitBC          |               maxwell,PIC,ImplicitO4               |                                                                         Implicit reflective particle BC                                                                          |    nProcs=1    | Particle Position |                                                        |
-|    7    |          [CHE_BGK](#che_bgkfpflow)          |                      BGK-Flow                      |                                                                                                                                                                                  |                |                   |                                                        |
-|    8    |        [CHE_FPFlow](#che_bgkfpflow)         |                      FP-Flow                       |                                                                                                                                                                                  |                |                   |                                                        |
+| **No.** |                   **Case**                  |                  **CMAKE-CONFIG**                  |                                                                                                **Feature**                                                                                               |  **Execution** |   **Comparing**   |                       **Readme**                       |
+| :-----: | :-----------------------------------------: | :------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------------: | :---------------: | :----------------------------------------------------: |
+|    1    |                  run_basic                  |                     maxwell,RK4                    |                                                                                                DG-Operator                                                                                               | nProcs=1,2,5,8 |      L2,Linf      |                                                        |
+|    2    |                 CHE_maxwell                 | maxwell, RK4, Particles=OFF, PICLAS_DEBUG_MEMORY=T |                                                                         DG-Operator, only Maxwell field solver, hard compiled N=1                                                                        | nProcs=1,2,5,8 |      L2,Linf      |                                                        |
+|    3    |             CHE_poisson/poisson             |                     Poisson,RK3                    | HDG-Operator, secondary electron emission (SEE-I model, does not happen because ions are too slow), hard compiled N=1, CalcBoundaryParticleOutput=T,CalcElectricTimeDerivative=T, UseH5IOLoadBalance=T,F | nProcs=1,2,5,8 |      L2,Linf      | [Link](/regressioncheck/CHE_poisson/poisson/readme.md) |
+|    4    | [CHE_PIC_maxwell_RK4](#che_pic_maxwell_rk4) |                 PIC (maxwell, RK4)                 |                                                                                         PIC-variableExternalField                                                                                        |                |                   |                                                        |
+|    5    |            [CHE_DSMC](#che_dsmc)            |                        DSMC                        |                                                                                                                                                                                                          |                |                   |                                                        |
+|    6    |          CHE_PIC_maxwell_implicitBC         |               maxwell,PIC,ImplicitO4               |                                                                                      Implicit reflective particle BC                                                                                     |    nProcs=1    | Particle Position |                                                        |
+|    7    |          [CHE_BGK](#che_bgkfpflow)          |                      BGK-Flow                      |                                                                                                                                                                                                          |                |                   |                                                        |
+|    8    |         [CHE_FPFlow](#che_bgkfpflow)        |                       FP-Flow                      |                                                                                                                                                                                                          |                |                   |                                                        |
 
 
 #### CHE_PIC_maxwell_RK4

diff --git a/docs/documentation/developerguide/mpi.md b/docs/documentation/developerguide/mpi.md
@@ -7,7 +7,7 @@ This chapter describes how PICLas subroutines and functions are parallelized.
 In case any new communicator (e.g. SurfCOMM%COMM) is built during init or anywhere else with
 `CALL MPI_COMM_SPLIT(NEWCOMMUNICATOR,iERROR)` or such, it is necessary to finalize it with `CALL MPI_COMM_FREE(NEWCOMMUNICATOR,iERROR)`.
 
-Else Loadbalances produce undefined errors that are almost impossible to find.
+Else, load balancing will produce undefined errors that are almost impossible to find.
 
 Debug MPI
 

diff --git a/docs/documentation/userguide/tutorials/dsmc-cone/dsmc-cone.md b/docs/documentation/userguide/tutorials/dsmc-cone/dsmc-cone.md
@@ -182,6 +182,10 @@ The parameter `Particles-MPIWeight` indicates whether the distribution should be
     LoadBalanceMaxSteps                      = 2
 
 Information about the imbalance are shown in the *std.out* and the *ElemTimeStatistics.csv* file.
+The default load balancing scheme will exchange the required data internally, but there is also the possibility to perform the
+re-balancing step via HDF5, which will create a state file and restart from this file by activating
+
+    UseH5IOLoadBalance = T ! default is False
 
 ## Visualization (post-processing)
 

diff --git a/regressioncheck/CHE_poisson/poisson/analyze.ini b/regressioncheck/CHE_poisson/poisson/analyze.ini
@@ -21,7 +21,7 @@ integrate_line_tolerance_type  = relative              ! relative or absolute
 ! compare the last row in SurfaceAnalyze.csv with a reference file
 compare_data_file_name           = SurfaceAnalyze.csv
 compare_data_file_reference      = SurfaceAnalyze_reference.csv
-compare_data_file_tolerance      = 60e-2 ! 1e-2
+compare_data_file_tolerance      = 80e-2 ! 1e-2
 compare_data_file_tolerance_type = relative
 
 ! ===================================================================================================================

diff --git a/regressioncheck/CHE_poisson/poisson/parameter.ini b/regressioncheck/CHE_poisson/poisson/parameter.ini
@@ -18,6 +18,7 @@ useCurveds    = F
 ! =============================================================================== !
 DoLoadBalance           = F
 DoInitialAutoRestart    = T
+UseH5IOLoadBalance      = T,F
 Load-DeviationThreshold = 1e-6
 LoadBalanceMaxSteps     = 1
 LoadBalanceSample       = 5

diff --git a/regressioncheck/CHE_poisson/poisson/readme.md b/regressioncheck/CHE_poisson/poisson/readme.md
@@ -18,3 +18,7 @@
 - Output of min/amax/average electron energy (eV) in each cell
 
         CalcElectronEnergy = T
+
+- Test both load balances schemes. Load balance via hdf5 I/O and via direct MPI communication without I/O
+
+        UseH5IOLoadBalance = T,F
diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/HDGIterations.csv b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/HDGIterations.csv
@@ -2,3 +2,25 @@
 0.0
 0.0
 0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
+0.0
diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/analyze.ini b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/analyze.ini
@@ -1,6 +1,13 @@
 analyze_L2 = 0
 
 ! compare columns in a data file
+! ----------------------------------------------------------------------------------------
+! IMPORTANT:
+! Every run with MPI=1,2,4,8 adds more lines to FieldAnalyze.csv, hence, the number of lines in
+! HDGIterations.csv must match (each line stands for a different run)
+! Note that CODE_ANALYZE=ON doubles the number of lines as a second FieldAnalyze is performed at restart for iter=0
+! Therefore 13 proc combinations x2 = 26 entries in HDGIterations.csv
+! ----------------------------------------------------------------------------------------
 compare_column_file            = post-piclas-restart/FieldAnalyze.csv ! data file name
 compare_column_reference_file  = HDGIterations.csv                    ! reference data file name
 compare_column_index           = 2                                    ! columns index (starts at 0)

diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/command_line.ini b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/command_line.ini
@@ -1 +1,3 @@
-MPI = 1 ! Create reference file with single execution
+! IMPORTANT: Create one reference file with single execution and run post-externals with multiple processors
+!            to test the restart functionality (0 iterations as the solution should be read in the correct way)
+MPI = 1
diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/externals.ini b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/externals.ini
@@ -1,13 +1,14 @@
 ! --- Externals Tool Reggie
 ! gcc/9.2.0   cmake/3.21.3  openmpi/3.1.6/gcc/9.2.0   hdf5/1.10.6/gcc/9.2.0/openmpi/3.1.6
-!MPI               = 1,2,3,4,5,6,7,8,9,10,11,12,13
+MPI               = 1,2,3,4,5,6,7,8,9,10,11,12,13
 
 ! gcc/11.2.0  cmake/3.21.3  openmpi/4.1.1/gcc/11.2.0  hdf5/1.12.1/gcc/11.2.0/openmpi/4.1.1
 ! Due to the change in compiler version, not all mpirun possibilities run without HDG iterations
-MPI               = 1,2,4,8
+! Update: this is wrong, there was a bug in the rotation of the DG (HDG) side solution
+!MPI               = 1,2,4,8
 
 externalbinary    = ./bin/piclas                                          ! Relative binary path in build directory
 externaldirectory = post-piclas-restart/                                  ! Directory name, where the files are located for the external tool reggie
 externalruntime   = post                                                  ! Run after piclas is completed (post: after, pre: before)
-cmd_suffix        = turner2013_State_000.00000000500000000.h5             ! Suffix for the binary execution
-cmd_pre_execute   = ln\s-sf\s../turner2013_State_000.00000000500000000.h5 ! "\s" resembles a white space character in the command (simply using " " is not allowed)
+cmd_suffix        = ../turner2013_State_000.00000000500000000.h5             ! Suffix for the binary execution
+!cmd_pre_execute   = ln\s-sf\s../turner2013_State_000.00000000500000000.h5 ! "\s" resembles a white space character in the command (simply using " " is not allowed)
diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/parameter.ini b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/parameter.ini
@@ -54,3 +54,41 @@ epsCG                 = 1e-6
 maxIterCG             = 10000
 HDGDisplayConvergence = T
 
+
+
+! XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+! Dummy parameter definitions for running with PARTICLES=ON
+! XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+! =============================================================================== !
+! PARTICLES
+! =============================================================================== !
+Part-maxParticleNumber=100000
+Part-nSpecies=1
+Part-nBounds=6
+Part-nPeriodicVectors=2
+
+Part-Boundary1-SourceName = BC_periodicz-
+Part-Boundary1-Condition  = periodic
+
+Part-Boundary2-SourceName = BC_periodicz+
+Part-Boundary2-Condition  = periodic
+
+Part-Boundary3-SourceName = BC_periodicy-
+Part-Boundary3-Condition  = periodic
+
+Part-Boundary4-SourceName = BC_periodicy+
+Part-Boundary4-Condition  = periodic
+
+Part-Boundary5-SourceName = BC_left
+Part-Boundary5-Condition  = reflective
+
+Part-Boundary6-SourceName = BC_right
+Part-Boundary6-Condition  = reflective
+
+Part-FIBGMdeltas=(/0.067,0.067,.067/)
+! =============================================================================== !
+! Species1 - O2
+! =============================================================================== !
+Part-Species1-MassIC=5.31352E-26
+Part-Species1-MacroParticleFactor=10
diff --git a/regressioncheck/NIG_poisson/poisson_box_Dirichlet/post-piclas-restart/parameter.ini b/regressioncheck/NIG_poisson/poisson_box_Dirichlet/post-piclas-restart/parameter.ini
@@ -32,7 +32,7 @@ CalcPotentialEnergy = T
 ! CALCULATION
 ! =============================================================================== !
 ManualTimestep  = 50.0E-10!1.0e-11 !4.58E-11!7.37463126843658E-010
-tend            = 50.0E-10
+tend            = 100.0E-10
 Analyze_dt      = 50.0E-10
 IterDisplayStep = 1
 
@@ -54,3 +54,41 @@ epsCG                 = 1e-6
 maxIterCG             = 10000
 HDGDisplayConvergence = T
 
+
+
+! XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+! Dummy parameter definitions for running with PARTICLES=ON
+! XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
+
+! =============================================================================== !
+! PARTICLES
+! =============================================================================== !
+Part-maxParticleNumber=100000
+Part-nSpecies=1
+Part-nBounds=6
+Part-nPeriodicVectors=2
+
+Part-Boundary1-SourceName = BC_periodicz-
+Part-Boundary1-Condition  = periodic
+
+Part-Boundary2-SourceName = BC_periodicz+
+Part-Boundary2-Condition  = periodic
+
+Part-Boundary3-SourceName = BC_periodicy-
+Part-Boundary3-Condition  = periodic
+
+Part-Boundary4-SourceName = BC_periodicy+
+Part-Boundary4-Condition  = periodic
+
+Part-Boundary5-SourceName = BC_left
+Part-Boundary5-Condition  = reflective
+
+Part-Boundary6-SourceName = BC_right
+Part-Boundary6-Condition  = reflective
+
+Part-FIBGMdeltas=(/0.067,0.067,.067/)
+! =============================================================================== !
+! Species1 - O2
+! =============================================================================== !
+Part-Species1-MassIC=5.31352E-26
+Part-Species1-MacroParticleFactor=10
diff --git a/regressioncheck/NIG_tracking_DSMC/ANSA_box/analyze.ini b/regressioncheck/NIG_tracking_DSMC/ANSA_box/analyze.ini
@@ -5,6 +5,7 @@ analyze_L2=1000
 h5diff_file            =           tildbox_State_001.00000000000000000.h5
 h5diff_reference_file  = tildbox_reference_State_001.0000000000000000.h5
 h5diff_data_set        = PartInt
+h5diff_flip            = T ! Transpose the array before comparing it (array has been flipped in the code but the old reference file is still being used)
 
 ! check if particles are outside of domain at tEnd
 check_hdf5_file        = tildbox_State_001.00000000000000000.h5

diff --git a/regressioncheck/NIG_tracking_DSMC/mortar/analyze.ini b/regressioncheck/NIG_tracking_DSMC/mortar/analyze.ini
@@ -5,6 +5,7 @@ analyze_L2=1000
 h5diff_file            =           Nonconform_State_000.10000000000000001.h5
 h5diff_reference_file  = Nonconform_reference_State_000.10000000000000001.h5
 h5diff_data_set        = PartInt
+h5diff_flip            = T ! Transpose the array before comparing it (array has been flipped in the code but the old reference file is still being used)
 
 ! check if particles are outside of domain at tEnd
 check_hdf5_file        = Nonconform_State_000.10000000000000001.h5

diff --git a/regressioncheck/NIG_tracking_DSMC/mortar_hexpress/analyze.ini b/regressioncheck/NIG_tracking_DSMC/mortar_hexpress/analyze.ini
@@ -4,3 +4,4 @@ h5diff_reference_file  = Mortar_hexpress_State_000.01000000000000000_ref.h5
 h5diff_data_set        = PartInt
 h5diff_tolerance_type  = relative
 h5diff_tolerance_value = 0.001
+h5diff_flip            = T ! Transpose the array before comparing it (array has been flipped in the code but the old reference file is still being used)
diff --git a/regressioncheck/NIG_tracking_DSMC/periodic/analyze.ini b/regressioncheck/NIG_tracking_DSMC/periodic/analyze.ini
@@ -5,6 +5,7 @@ analyze_L2=1000
 h5diff_file            =           periodic_State_000.02000000000000000.h5
 h5diff_reference_file  = periodic_reference_State_000.0200000000000000.h5
 h5diff_data_set        = PartInt
+h5diff_flip            = T ! Transpose the array before comparing it (array has been flipped in the code but the old reference file is still being used)
 
 ! check if particles are outside of domain at tEnd
 check_hdf5_file        = periodic_State_000.02000000000000000.h5
-Original file line number
+Diff line change
@@ Expand Up / @@ -2,3 +2,25 @@ @@
 .0
 .0
 .0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0
+.0