Using mass_inflow_outflow with multiple refinement levels

[lawrenceccheung]

Bug description

I am using the mass_inflow_outflow boundary condition on a case with multiple levels of refinement, although no refinement levels are touching the boundaries, and the boundary data is captured with level 0 only.

Running the case encounters an error on the first step:

terminate called after throwing an instance of 'std::runtime_error'
  what():  Assertion `outflux > eps' failed, file "/projects/wind_uq/lcheung/AMRWindBuilds/hfm.20250123/amr-wind/submods/AMReX-Hydro/Utils/hydro_enforce_inout_solvability.cpp", line 323, Msg: "Cannot enforce solvability, no outflow from the direction dependent boundaries"
[flight561:67906] *** Process received signal ***
[flight561:67906] Signal: Aborted (6)
[flight561:67906] Signal code:  (-6)

This points to a problem in
https://github.com/AMReX-Fluids/AMReX-Hydro/blob/development/Utils/hydro_enforce_inout_solvability.cpp#L322, although it might also be prevented from the AMR-Wind side.

Steps to reproduce

The full case set up details are at
https://github.com/lawrenceccheung/AWAKEN_summit_setup/blob/main/Benchmark1/Phase2/FarmRuns/prod1/BM1Ph2_FarmRunProd1newBC.ipynb

Quick summary:

• mass_inflow_outflow is used on all 4 sides (xlo, ylo, xhi, yhi) due to highly turbulent eddies going into and out of each face. Predominant flow direction is changing, but generally around 220-240 degrees.
• Meso/microscale forcing is used in this case, but I don't think that should affect anything.
• Refinement zones are shown here:
  

Note:

• Case works perfectly if amr.max_level is set to 0.
• You can also make this case work if we change this loop in hydro_enforce_inout_solvability.cpp to only loop over level 0.

    const auto nlevs = int(vels_vec.size());
    for (int lev = 0; lev < nlevs; ++lev) {
    }

Steps to reproduce the behavior:

1. Compiler used

   • GCC
   • LLVM
   • oneapi (Intel)
   • nvcc (NVIDIA)
   • rocm (AMD)
   • with MPI
   • other:

2. Operating system

   • Linux
   • OSX
   • Windows
   • other (do tell ;)):

3. Hardware:

   • CPU
   • GPU

4. Machine details ():
   Sandia flight on 7168 ranks.

5. Input file attachments: See setup notebook above. I am also working on a smaller version of this case for debugging.

6. Error (paste or attach):
   See sbove.

7. If this is a segfault, a stack trace from a debug build (paste or attach):
   See above.

Expected behavior

mass_inflow_outflow should be usable with multiple refinement levels.

AMR-Wind information

==============================================================================
                AMR-Wind (https://github.com/exawind/amr-wind)

  AMR-Wind version :: v3.3.1-44-gd7046e1c
  AMR-Wind Git SHA :: d7046e1c70246f5c461f3244bc85958b60398310
  AMReX version    :: 25.01-16-g92d35c2c8163

  Exec. time       :: Thu Feb  6 20:11:43 2025
  Build time       :: Feb  3 2025 10:57:50
  C++ compiler     :: GNU 12.1.0

  MPI              :: ON    (Num. ranks = 7168)
  GPU              :: OFF
  OpenMP           :: OFF

  Enabled third-party libraries: 
    NetCDF    4.9.2
    OpenFAST  

           This software is released under the BSD 3-clause license.           
 See https://github.com/Exawind/amr-wind/blob/development/LICENSE for details. 
------------------------------------------------------------------------------

Initializing AMReX (25.01-16-g92d35c2c8163)...

[hgopalan]

I noticed that eps is set to 1e-6 explicitly on top of the code. First check may be to print the outflux value and check if tolerance has to be made user-specified. The test cases, I ran were initialized and setup differently from yours so my comment on refinement levels working fine may not apply here.

[marchdf]

@mukul1992 and @asalmgren do you have thoughts on this? I think this came from work you did with the inflow/outflow stuff. Any tips to debug would be most welcome.

[mukul1992]

I think I have a hunch of what might be wrong. This routine scales the outflow to match the inflow so that the projection is solvable. If there is no outflow, there is nothing to scale, so the above assertion checks for that.

But in the above case where the finer level is not touching the boundary at all, it is okay for the outflow to be zero as far as the inflow is zero.

So maybe it should be something like this:

        if ((influx > eps) && (outflux < eps)) {
            Abort("Cannot enforce solvability, no outflow from the direction dependent boundaries");
        } else if ((influx < eps) && (outflux < eps)) {
            return; // do nothing
        } else {
            const Real alpha_fcf = influx/outflux;  // flux correction factor
            correct_outflow(lev, vels_vec, inout_masks, bc_type, domain, alpha_fcf, include_bndry_corners);
        }

I'll ask Ann about this solution, meanwhile feel free to try it out.

[lawrenceccheung]

Thanks @hgopalan, @mukul1992! I'm creating a smaller test case which has the same flow properties, but will allow me to test out various fixes faster. I'll give these a try.

Lawrence

[mukul1992]

FYI the relevant PR for Hydro is up, waiting to be merged and then updated in AMR-Wind via the submodule.

[mukul1992]

Update: the PR is now merged into Hydro.

[marchdf]

Fantastic work @mukul1992! PR for updating the submodule is here: #1483.

@lawrenceccheung once you confirm that works, I will merge.

[marchdf]

It got merged and closed this issue. So I am reopening it until @lawrenceccheung can confirm the fix is good.

[sbidadi9]

@mukul1992 @marchdf @hgopalan I'm running with this boundary condition together with cloud data being injected at the inflow boundary. However, I get the same error as Lawrence. The case can be found here: /scratch/sbidadi/oracle/oracle_sims/bc_code/case/cloud_sim_dt_fixed_abl_forcing_corr_init_condition_fine_mesh_ar_1/cloud_dx_10_wenoz_io_bc

cc: @mchurchf

[moprak-nrel]

Just to double check, you're running this with the updated submod?

[lawrenceccheung]

Hi all,
I can confirm that using amr-wind with the AMReX-Hydro submodule 40b0975 (the one with @mukul1992's update) works to solve the problem I was having.

Interestingly, if it's just a little tiny refinement box in the middle of the domain, this problem with mass_inflow_outflow does not come up:

However, once you start expanding the box so that some of the blocking factors might expand the refinement zone to be in the same general vicinity of the boundary:

then this becomes a problem, and the fix to AMReX-Hydro becomes necessary.

Also a side note -- using mass_inflow_outflow also seems to affect the linear solver. I'm seeing convergence problems with nodal projections, but let me start a new issue to work on that.

Lawrence