HPMG: Compute finest level directly on the field #1137
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AlexanderSinn
added
component: fields
|
On A100 with 4095^2 cells, this provides a 22% performance improvement. @WeiqunZhang could there be accuracy issues from doing gsrb directly on the sol array instead of taking the residual first? |
|
If it converges, I guess it's fine. |
|
I still need to check if setting the Bx By boundary condition with MR still works the same or if it has to be different now. |
…ps://github.com/AlexanderSinn/hipace into HPMG_compute_finest_level_directly_on_the_field
MaxThevenet
approved these changes
Jul 31, 2024
Comment on lines
-728
to
+745
| if (lev!=0 && (slicemf.box(0).length(0) % 2 == 0)) { | ||
| // cell centered MG solve: no ghost cells, put boundary condition into source term | ||
| // node centered MG solve: one ghost cell, use boundary condition from there | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "Bx", | ||
| m_fields.getField(lev, which_slice, "Sy"), | ||
| amrex::IntVect{0, 0, 0}, 0.5, 8./3.); | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "By", | ||
| m_fields.getField(lev, which_slice, "Sx"), | ||
| amrex::IntVect{0, 0, 0}, 0.5, 8./3.); | ||
| if (lev!=0) { | ||
| if (slicemf.box(0).length(0) % 2 == 0) { | ||
| // cell centered MG solve: | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "Bx", | ||
| m_fields.getField(lev, which_slice, "Sy"), | ||
| amrex::IntVect{0, 0, 0}, 0.5, 8./3.); | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "By", | ||
| m_fields.getField(lev, which_slice, "Sx"), | ||
| amrex::IntVect{0, 0, 0}, 0.5, 8./3.); | ||
| } else { | ||
| // node centered MG solve: | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "Bx", | ||
| m_fields.getField(lev, which_slice, "Sy"), | ||
| amrex::IntVect{0, 0, 0}, 1., 1.); | ||
| m_fields.SetBoundaryCondition(m_3D_geom, lev, which_slice, "By", | ||
| m_fields.getField(lev, which_slice, "Sx"), | ||
| amrex::IntVect{0, 0, 0}, 1., 1.); | ||
| } |
There was a problem hiding this comment.
Could you say a few words on this change?
Comment on lines
+823
to
+828
|
|
||
| // interpolate Bx and By to lev from lev-1 in the ghost cells | ||
| m_fields.LevelUpBoundary(m_3D_geom, lev, which_slice, "Bx", | ||
| m_fields.m_slices_nguards, m_fields.m_poisson_nguards); | ||
| m_fields.LevelUpBoundary(m_3D_geom, lev, which_slice, "By", | ||
| m_fields.m_slices_nguards, m_fields.m_poisson_nguards); |
There was a problem hiding this comment.
Because the nodal MG solver doesn't read in the ghost cells anymore, the level >=1 boundary condition has to come from SetBoundaryCondition instead of LevelUpBoundary.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Quite a bit of time in HPMG is spent copying phi between the HPMG array and field array, as the coarsest level has the largest resolution. This PR changes the vcycle so that the finest level is partly directly calculated on the input field array, instead of computing the residual first and then starting at zero. The gsrb_4_residual function is now also used to compute the residual to determine if there should be another vcycle instead of compute_residual, this way gsrb_4_residual can be omitted at the start of the vcycle. This will result in four extra gsrb iterations being done on the finest level, slightly changing CI test results. Additionally, this PR includes some minor cleaning and other smaller HPMG performance improvements.
When using tighter tolerances, the PR and development MG solvers are closer to each other than to the checksum with the lower tolerance (the changes made by this PR are within the tolerance of the MG solver). See
https://github.com/Hi-PACE/hipace/actions/runs/10078089809/job/27862184203?pr=1137
https://github.com/Hi-PACE/hipace/actions/runs/10077289597/job/27859470194?pr=1139
constisconst)