Air-Sea CO₂ Flux dataset throws 500 on the /statistics endpoint

[dzole0311]

I noticed that choosing Delaware, Arizona, Indiana and a few other presets and running the analysis, the Air-Sea CO₂ Flux dataset errors out with a 500 on the statistics endpoint. It works though if I choose California as a preset.

It can be reproduced here.

Maybe related: 232

[j08lue]

@amarouane-ABDELHAK or @slesaad - would you be able to check what is going wrong in the backend here, please?

[slesaad]

@j08lue will do!

[slesaad]

The errors we're getting in the titiler API lambda:

GDAL signalled an error: err_no=1, msg='PROJ: cea: Invalid latitude'

and

ZeroDivisionError: Weights sum to zero, can't be normalized

trace

[ERROR]	2024-05-29T18:50:59.623Z	d5663d57-9e91-40f7-b656-37761227c4bb	An error occurred running the application.
Traceback (most recent call last):
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 98, in receive
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 93, in receive_nowait
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 78, in call_next
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/streams/memory.py", line 118, in receive
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/pip-target-r4n_td5g/lib/python/mangum/protocols/http.py", line 66, in run
  File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/applications.py", line 1054, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/applications.py", line 122, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/errors.py", line 184, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/errors.py", line 162, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 108, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/src/app.py", line 195, in add_correlation_id
  File "/tmp/pip-target-r4n_td5g/lib/python/aws_lambda_powertools/tracing/tracer.py", line 581, in decorate
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 84, in call_next
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/base.py", line 70, in coro
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette_cramjam/middleware.py", line 112, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette_cramjam/middleware.py", line 142, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/titiler/core/middleware.py", line 63, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/cors.py", line 83, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/exceptions.py", line 79, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/middleware/exceptions.py", line 68, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 718, in __call__
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 276, in handle
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/routing.py", line 66, in app
  File "/tmp/pip-target-r4n_td5g/lib/python/src/monitoring.py", line 41, in route_handler
  File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 299, in app
  File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 294, in app
  File "/tmp/pip-target-r4n_td5g/lib/python/fastapi/routing.py", line 193, in run_endpoint_function
  File "/tmp/pip-target-r4n_td5g/lib/python/starlette/concurrency.py", line 41, in run_in_threadpool
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/to_thread.py", line 33, in run_sync
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
  File "/tmp/pip-target-r4n_td5g/lib/python/anyio/_backends/_asyncio.py", line 807, in run
  File "/tmp/pip-target-r4n_td5g/lib/python/titiler/core/factory.py", line 490, in geojson_statistics
  File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/models.py", line 776, in statistics
  File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/utils.py", line 191, in get_array_statistics
  File "/tmp/pip-target-r4n_td5g/lib/python/rio_tiler/utils.py", line 52, in _weighted_stdev
  File "/tmp/pip-target-r4n_td5g/lib/python/numpy/lib/function_base.py", line 550, in average
ZeroDivisionError: Weights sum to zero, can't be normalized

@vincentsarago might need your intervention here

Request URL:
https://ghg.center/api/raster/cog/statistics?url=s3:%2F%2Fghgc-data-store%2Feccodarwin-co2flux-monthgrid-v5%2FECCO-Darwin_CO2_flux_202211.tif&nodata=nan&dst_crs=%2Bproj%3Dcea

body:

{
  "type": "Feature",
  "properties": {},
  "geometry": {
    "type": "MultiPolygon",
    "coordinates": [
      [
        [
          [
            -75.78859,
            39.7222
          ],
          [
            -75.7669,
            39.37765
          ],
          [
            -75.76689,
            39.3775
          ],
          [
            -75.76044,
            39.29679
          ],
          [
            -75.75643,
            39.24669
          ],
          [
            -75.74815,
            39.14313
          ],
          [
            -75.7231,
            38.82983
          ],
          [
            -75.70755,
            38.63539
          ],
          [
            -75.70755,
            38.63534
          ],
          [
            -75.70177,
            38.56077
          ],
          [
            -75.70038,
            38.54274
          ],
          [
            -75.69372,
            38.46013
          ],
          [
            -75.47928,
            38.4537
          ],
          [
            -75.34128,
            38.45244
          ],
          [
            -75.18545,
            38.45102
          ],
          [
            -75.04893,
            38.45127
          ],
          [
            -75.05397,
            38.53628
          ],
          [
            -75.0718,
            38.6965
          ],
          [
            -75.08947,
            38.7972
          ],
          [
            -75.11333,
            38.783
          ],
          [
            -75.15902,
            38.7902
          ],
          [
            -75.23203,
            38.84426
          ],
          [
            -75.30408,
            38.91316
          ],
          [
            -75.30255,
            38.939
          ],
          [
            -75.30665,
            38.94766
          ],
          [
            -75.34089,
            39.01996
          ],
          [
            -75.39628,
            39.05789
          ],
          [
            -75.40747,
            39.13371
          ],
          [
            -75.39479,
            39.18836
          ],
          [
            -75.40837,
            39.2647
          ],
          [
            -75.46932,
            39.33082
          ],
          [
            -75.50564,
            39.3704
          ],
          [
            -75.52168,
            39.38787
          ],
          [
            -75.57183,
            39.4389
          ],
          [
            -75.59307,
            39.47919
          ],
          [
            -75.52809,
            39.49812
          ],
          [
            -75.52767,
            39.53528
          ],
          [
            -75.51273,
            39.578
          ],
          [
            -75.54396,
            39.596
          ],
          [
            -75.55944,
            39.62981
          ],
          [
            -75.53514,
            39.64721
          ],
          [
            -75.50974,
            39.68611
          ],
          [
            -75.47764,
            39.71501
          ],
          [
            -75.45944,
            39.76581
          ],
          [
            -75.41506,
            39.80192
          ],
          [
            -75.48121,
            39.82919
          ],
          [
            -75.57043,
            39.83919
          ],
          [
            -75.57965,
            39.83742
          ],
          [
            -75.59256,
            39.83493
          ],
          [
            -75.59432,
            39.8346
          ],
          [
            -75.66284,
            39.82143
          ],
          [
            -75.71706,
            39.79233
          ],
          [
            -75.75323,
            39.75799
          ],
          [
            -75.77378,
            39.7222
          ],
          [
            -75.78859,
            39.7222
          ]
        ]
      ]
    ]
  }
}

This is the file:

ECCO-Darwin_CO2_flux_202211.tif.zip

[j08lue]

Great report, @slesaad, thanks!

The /statistics request currently succeeds for me:

{
"b1": {
"min": -0.031021814920656237,
"max": 0.0307270460145791,
"mean": -0.000021450558869718596,
"count": 338606,
"sum": -7.263287936639935,
"std": 0.000443675030901694,
"median": -0.00000347941618740146,
"majority": -0.031021814920656237,
"minority": -0.031021814920656237,
"unique": 338606,
"histogram": [
[
9,
0,
0,
0,
19345,
319192,
0,
0,
0,
60
],
[
-0.031021814920656237,
-0.024846928827132703,
-0.01867204273360917,
-0.012497156640085636,
-0.0063222705465621025,
-0.00014738445303856887,
0.006027501640484965,
0.012202387734008498,
0.018377273827532032,
0.024552159921055566,
0.0307270460145791
]
],
"valid_percent": 64.46,
"masked_pixels": 186706,
"valid_pixels": 338606,
"percentile_2": -0.00018335218910390857,
"percentile_98": 0.00006901845370872633
}
}

But when looking at the /info, I see what GDAL seems to be complaining about in your traceback above:

{
"bounds": [
-180.125,
-90.124826629681,
179.875,
89.875173370319
],
"minzoom": 0,
"maxzoom": 2,
"band_metadata": [
[
"b1",
{}
]
],
"band_descriptions": [
[
"b1",
"Air-sea CO2 Flux"
]
],
"dtype": "float64",
"nodata_type": "None",
"colorinterp": [
"gray"
],
"driver": "GTiff",
"count": 1,
"width": 1440,
"height": 721,
"overviews": [
2,
4
]
}

i.e. latitudes ranging from -90.124826629681 to 89.875173370319. -90.12 is obviously out of bounds. We should check whether something went wrong during the creation of the ECCO-Darwin files.

[j08lue]

• https://github.com/US-GHG-Center/ghgc-data/issues/20

[SwordSaintLancelot]

This issue was caused by selecting the wrong area on the map. @j08lue already mentioned that we do get the statistics for the COGs. Can we close this issue as it seems to be resolved? @slesaad

[slesaad]

@SwordSaintLancelot since we have another ticket for wrong bounds, we can close this