Water is not conserved when precipitation is present

[szy21]

The simplest case to reproduce this is possibly enabling PrognosticSurfaceTemperature is the single_column_precipitation_test case. Steps to reproduce:

import ClimaAtmos as CA
import SciMLBase
config = CA.AtmosConfig("configs/single_column_precipitation_test.yml")
config.parsed_args["prognostic_surface"] = "PrognosticSurfaceTemperature"
simulation = CA.get_simulation(config)
initial_water = sum(simulation.integrator.u.c.ρq_tot)
SciMLBase.step!(simulation.integrator)
water_sfc = CA.horizontal_integral_at_boundary(simulation.integrator.u.sfc.water)
water_tot = sum(simulation.integrator.u.c.ρq_tot)

println((water_sfc + water_tot - initial_water)/ water_tot)

This should be round-off, but it is not.

We verified that water is well conserved in absence of precipitation.

Investigation started by

See these two builds with ClimaAtmos v0.26.0:
ARS: https://buildkite.com/clima/climaatmos-gpulongruns/builds/245#01903269-239f-4966-8cc9-f7d6dc2c2570/155-574
net water change: 5.4187683e-5
SSP: https://buildkite.com/clima/climaatmos-gpulongruns/builds/244#019031bf-c8a3-4be3-861e-6eb54f165278/155-576
net water change: 0.23737198 (!)

@dennisYatunin @Sbozzolo Could you take a look if the test is wrong, or if there is something with SSP?

[Sbozzolo]

Yes

[Sbozzolo]

Some updates:

• ClimaAtmos main with ARS343 has a 7x worse conversation than the job posted in OP: https://buildkite.com/clima/climaatmos-gpulongruns/builds/260#01905532-7a40-4935-bb77-e53b1d4a0ac4
• SSPKnoth and ARS343 show the same conversation properties for Akshay's moist bubble case (periodic box, no PrognosticSurfaceTemperature)
• SSPKnoth show conservation at round-off error in absence of rain, and starts getting worse when it start raining
• ARS343 starts with not great conservation, but it improves with time`
• The problem persists with Float64
• When changing vert_diff to true, even ARS343 shows poor water conservation, with error of order > 2%