`CUDA error: ERROR_ILLEGAL_ADDRESS` when running the `three_layer_constant_fluxes.jl` script #117

ali-ramadhan · 2021-05-26T13:30:30Z

Will see if I can isolate the cause...

ERROR: LoadError: CUDA error: an illegal memory access was encountered (code 700, ERROR_ILLEGAL_ADDRESS)
Stacktrace:
  [1] throw_api_error(res::CUDA.cudaError_enum)
    @ CUDA ~/.julia/packages/CUDA/nbJRs/lib/cudadrv/error.jl:105
  [2] macro expansion
    @ ~/.julia/packages/CUDA/nbJRs/lib/cudadrv/error.jl:115 [inlined]
  [3] cuCtxSynchronize()
    @ CUDA ~/.julia/packages/CUDA/nbJRs/lib/utils/call.jl:26
  [4] device_synchronize
    @ ~/.julia/packages/CUDA/nbJRs/lib/cudadrv/context.jl:310 [inlined]
  [5] CUDA.CuModule(data::String, options::Dict{CUDA.CUjit_option_enum, Any})
    @ CUDA ~/.julia/packages/CUDA/nbJRs/lib/cudadrv/module.jl:41
  [6] cufunction_link(job::GPUCompiler.CompilerJob, compiled::NamedTuple{(:code, :entry, :needs_cudadevrt, :external_gvars), Tuple{String, String, Bool, Vector{String}}})
    @ CUDA ~/.julia/packages/CUDA/nbJRs/src/compiler/execution.jl:348
  [7] check_cache
    @ ~/.julia/packages/GPUCompiler/eJOtJ/src/cache.jl:56 [inlined]
  [8] cached_compilation
    @ ./none:0 [inlined]
  [9] cufunction(f::typeof(Cassette.overdub), tt::Type{Tuple{Cassette.Context{nametype(CUDACtx), KernelAbstractions.CompilerMetadata{KernelAbstractions.NDIteration.StaticSize{(32, 32, 32)}, KernelAbstractions.NDIteration.DynamicCheck, Nothing, Nothing, KernelAbstractions.NDIteration.NDRange{3, KernelAbstractions.NDIteration.StaticSize{(2, 2, 32)}, KernelAbstractions.NDIteration.StaticSize{(16, 16, 1)}, Nothing, Nothing}}, Nothing, KernelAbstractions.var"##PassType#257", Nothing, Cassette.DisableHooks}, typeof(Oceananigans.Models.IncompressibleModels.gpu_calculate_Gv!), OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, WENO5, FPlane{Float64}, Nothing, AnisotropicMinimumDissipation{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Nothing}, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Float64}, Oceananigans.BuoyancyModels.ZDirection}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Oceananigans.Fields.ZeroField}}}}, NamedTuple{(:u, :v, :w), Tuple{OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}}}}}, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Nothing, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Nothing, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Nothing, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, LinearTarget{:z, Float64}}, Nothing, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Nothing, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Nothing, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Nothing, Tuple{}, Tuple{}}}}, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuDeviceArray{Float64, 3, 1}}, NamedTuple{(:time, :iteration, :stage), Tuple{Float64, Int64, Int64}}}}; name::String, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ CUDA ~/.julia/packages/CUDA/nbJRs/src/compiler/execution.jl:288
 [10] macro expansion
    @ ~/.julia/packages/CUDA/nbJRs/src/compiler/execution.jl:102 [inlined]
 [11] (::KernelAbstractions.Kernel{CUDAKernels.CUDADevice, KernelAbstractions.NDIteration.StaticSize{(16, 16)}, KernelAbstractions.NDIteration.StaticSize{(32, 32, 32)}, typeof(Oceananigans.Models.IncompressibleModels.gpu_calculate_Gv!)})(::Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, ::Vararg{Any, N} where N; ndrange::Nothing, dependencies::CUDAKernels.CudaEvent, workgroupsize::Nothing, progress::Function)
    @ CUDAKernels ~/.julia/packages/CUDAKernels/8wtKq/src/CUDAKernels.jl:194
 [12] calculate_interior_tendency_contributions!(tendencies::NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, arch::GPU, grid::RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, advection::WENO5, coriolis::FPlane{Float64}, buoyancy::Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Float64}, Oceananigans.BuoyancyModels.ZDirection}, stokes_drift::Nothing, closure::AnisotropicMinimumDissipation{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Nothing}, background_fields::NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Oceananigans.Fields.ZeroField}}}}, velocities::NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}}}, tracers::NamedTuple{(:T, :c₀, :c₁, :c₂), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Gradient, Float64}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, hydrostatic_pressure::Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, diffusivities::NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}}}, forcings::NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, LinearTarget{:z, Float64}}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}}}, clock::Clock{Float64})
    @ Oceananigans.Models.IncompressibleModels ~/.julia/packages/Oceananigans/IxOwr/src/Models/IncompressibleModels/calculate_tendencies.jl:82
 [13] calculate_tendencies!
    @ ~/.julia/packages/Oceananigans/IxOwr/src/Models/IncompressibleModels/calculate_tendencies.jl:24 [inlined]
 [14] time_step!(model::IncompressibleModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, Nothing}, AnisotropicMinimumDissipation{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Nothing}, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Float64}, Oceananigans.BuoyancyModels.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Gradient, Float64}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, LinearTarget{:z, Float64}}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}}}, WENO5, Oceananigans.Solvers.FFTBasedPoissonSolver{GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:λx, :λy, :λz), Tuple{CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}}}, CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}, Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}}}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing}, Δt::Float64)
    @ Oceananigans.TimeSteppers ~/.julia/packages/Oceananigans/IxOwr/src/TimeSteppers/runge_kutta_3.jl:82
 [15] #ab2_or_rk3_time_step!#6
    @ ~/.julia/packages/Oceananigans/IxOwr/src/Simulations/run.jl:69 [inlined]
 [16] run!(sim::Simulation{IncompressibleModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, Nothing}, AnisotropicMinimumDissipation{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Nothing}, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Float64}, Oceananigans.BuoyancyModels.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Gradient, Float64}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, LinearTarget{:z, Float64}}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}}}, WENO5, Oceananigans.Solvers.FFTBasedPoissonSolver{GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:λx, :λy, :λz), Tuple{CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}}}, CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}, Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}}}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing}, TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, Vector{Any}, Float64, Float64, Float64, Float64, OrderedCollections.OrderedDict{Symbol, Oceananigans.AbstractDiagnostic}, OrderedCollections.OrderedDict{Symbol, Oceananigans.AbstractOutputWriter}, SimulationProgressMessenger{Float64, CFL{TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, typeof(cell_advection_timescale)}, CFL{TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, typeof(Oceananigans.TurbulenceClosures.cell_diffusion_timescale)}, TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}}, Int64, Nothing}; pickup::Bool)
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/IxOwr/src/Simulations/run.jl:177
 [17] run!(sim::Simulation{IncompressibleModel{Oceananigans.TimeSteppers.RungeKutta3TimeStepper{Float64, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, Nothing}, AnisotropicMinimumDissipation{Oceananigans.TurbulenceClosures.ExplicitTimeDiscretization, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Float64, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Float64}}, Nothing}, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, Buoyancy{SeawaterBuoyancy{Float64, LinearEquationOfState{Float64}, Nothing, Float64}, Oceananigans.BuoyancyModels.ZDirection}, FPlane{Float64}, Nothing, NamedTuple{(:u, :v, :w), Tuple{Field{Face, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Face, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Face, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{NormalFlow, Nothing}, BoundaryCondition{NormalFlow, Nothing}}}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Gradient, Float64}, BoundaryCondition{Flux, Float64}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:pHY′, :pNHS), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}, NamedTuple{(:u, :v, :w, :T, :c₀, :c₁, :c₂), Tuple{Oceananigans.Forcings.ContinuousForcing{Face, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity1)}}, Oceananigans.Forcings.ContinuousForcing{Center, Face, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity2)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Face, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, typeof(Oceananigans.Forcings.zerofunction)}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity3)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, Nothing, Relaxation{Float64, GaussianMask{:z, Float64}, LinearTarget{:z, Float64}}, Tuple{Symbol}, Tuple{Int64}, Tuple{typeof(Oceananigans.Operators.identity4)}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}, Oceananigans.Forcings.ContinuousForcing{Center, Center, Center, NamedTuple{(:z₀, :λ, :μ⁺, :μ⁻), NTuple{4, Float64}}, typeof(passive_tracer_forcing), Tuple{}, Tuple{}, Tuple{}}}}, WENO5, Oceananigans.Solvers.FFTBasedPoissonSolver{GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, NamedTuple{(:λx, :λy, :λz), Tuple{CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}, CUDA.CuArray{Float64, 3}}}, CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}, NamedTuple{(:forward, :backward), Tuple{Tuple{Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}, Oceananigans.Solvers.DiscreteTransform{CUDA.CUFFT.cCuFFTPlan{ComplexF64, -1, true, 3}, Oceananigans.Solvers.Forward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}}, Tuple{Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Vector{Int64}, Vector{Periodic}, Int64, Nothing, Nothing}, Oceananigans.Solvers.DiscreteTransform{AbstractFFTs.ScaledPlan{ComplexF64, CUDA.CUFFT.cCuFFTPlan{ComplexF64, 1, true, 3}, Float64}, Oceananigans.Solvers.Backward, GPU, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Int64, Bounded, Int64, NamedTuple{(:forward, :backward), Tuple{CUDA.CuArray{ComplexF64, 3}, CUDA.CuArray{ComplexF64, 3}}}, Nothing}}}}}, NamedTuple{(:νₑ, :κₑ), Tuple{Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Field{Center, Center, Center, GPU, OffsetArrays.OffsetArray{Float64, 3, CUDA.CuArray{Float64, 3}}, RegularRectilinearGrid{Float64, Periodic, Periodic, Bounded, OffsetArrays.OffsetVector{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}}}}, Float64, NamedTuple{(:x, :y, :z), Tuple{CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}, BoundaryCondition{Oceananigans.BoundaryConditions.Periodic, Nothing}}, CoordinateBoundaryConditions{BoundaryCondition{Flux, Nothing}, BoundaryCondition{Flux, Nothing}}}}}}}}}, NamedTuple{(:velocities, :tracers), Tuple{NamedTuple{(:u, :v, :w), Tuple{Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField, Oceananigans.Fields.ZeroField}}, NamedTuple{(:T, :c₀, :c₁, :c₂), NTuple{4, Oceananigans.Fields.ZeroField}}}}, Nothing, Nothing}, TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, Vector{Any}, Float64, Float64, Float64, Float64, OrderedCollections.OrderedDict{Symbol, Oceananigans.AbstractDiagnostic}, OrderedCollections.OrderedDict{Symbol, Oceananigans.AbstractOutputWriter}, SimulationProgressMessenger{Float64, CFL{TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, typeof(cell_advection_timescale)}, CFL{TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}, typeof(Oceananigans.TurbulenceClosures.cell_diffusion_timescale)}, TimeStepWizard{Float64, typeof(cell_advection_timescale), typeof(Oceananigans.Simulations.infinite_diffusion_timescale)}}, Int64, Nothing})
    @ Oceananigans.Simulations ~/.julia/packages/Oceananigans/IxOwr/src/Simulations/run.jl:127
 [18] top-level scope
    @ ~/LESbrary.jl/examples/three_layer_constant_fluxes.jl:575
 [19] include(fname::String)
    @ Base.MainInclude ./client.jl:444
 [20] top-level scope
    @ REPL[2]:1
in expression starting at /home/alir/LESbrary.jl/examples/three_layer_constant_fluxes.jl:575

The text was updated successfully, but these errors were encountered:

ali-ramadhan · 2021-05-26T16:01:31Z

Script compiles and runs without the AMD closure it seems (see diff below) but no idea why.

Trying to reproduce in vanilla Oceananigans.jl but no luck so far as it doesn't even precompile: CliMA/Oceananigans.jl#1701

diff --git a/examples/three_layer_constant_fluxes.jl b/examples/three_layer_constant_fluxes.jl
index 042466d..1a0aedd 100644
--- a/examples/three_layer_constant_fluxes.jl
+++ b/examples/three_layer_constant_fluxes.jl
@@ -293,7 +293,7 @@
model = IncompressibleModel(
                 tracers = (:T, :c₀, :c₁, :c₂),
                buoyancy = buoyancy,
                coriolis = FPlane(f=f),
-                closure = AnisotropicMinimumDissipation(),
+              # closure = AnisotropicMinimumDissipation(),
     boundary_conditions = (T=θ_bcs, u=u_bcs),
                 forcing = (u=u_sponge, v=v_sponge, w=w_sponge, T=T_sponge,
                            c₀=c₀_forcing, c₁=c₁_forcing, c₂=c₂_forcing)

+ @info "Try taking a time step..."
+ time_step!(model, 0.1)

ali-ramadhan · 2021-05-26T22:06:54Z

@xkykai mentioned that he does not get this error on Supercloud with the same version of packages but with CUDA 11.x with a V100. So the error might be CUDA driver version dependent (possibly also GPU dependent?).

ali-ramadhan mentioned this issue May 26, 2021

Try latest CUDA.jl CliMA/Oceananigans.jl#1701

Closed

ali-ramadhan mentioned this issue May 27, 2021

Hacky horizontal averages and Julia 1.5+ support #119

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`CUDA error: ERROR_ILLEGAL_ADDRESS` when running the `three_layer_constant_fluxes.jl` script #117

`CUDA error: ERROR_ILLEGAL_ADDRESS` when running the `three_layer_constant_fluxes.jl` script #117

ali-ramadhan commented May 26, 2021

ali-ramadhan commented May 26, 2021 •

edited

Loading

ali-ramadhan commented May 26, 2021

CUDA error: ERROR_ILLEGAL_ADDRESS when running the three_layer_constant_fluxes.jl script #117

CUDA error: ERROR_ILLEGAL_ADDRESS when running the three_layer_constant_fluxes.jl script #117

Comments

ali-ramadhan commented May 26, 2021

ali-ramadhan commented May 26, 2021 • edited Loading

ali-ramadhan commented May 26, 2021

`CUDA error: ERROR_ILLEGAL_ADDRESS` when running the `three_layer_constant_fluxes.jl` script #117

`CUDA error: ERROR_ILLEGAL_ADDRESS` when running the `three_layer_constant_fluxes.jl` script #117

ali-ramadhan commented May 26, 2021 •

edited

Loading