agent: ClientResponseError: Bad Gateway when getting list of releases

[sentry-io]

Sentry Issue: BALROG-PROD-AGENT-2

ClientResponseError: Bad Gateway
  File "cmd.py", line 81, in run_agent
    await process_v2_scheduled_changes(loop, balrog_api_root, auth0_secrets)
  File "cmd.py", line 68, in process_v2_scheduled_changes
    resp = await client.request(balrog_api_root, endpoint, loop=loop, auth0_secrets=auth0_secrets)
  File "balrogagent/client.py", line 55, in request
    resp.raise_for_status()
  File "aiohttp/client_reqrep.py", line 1129, in raise_for_status
    raise ClientResponseError(

Encountered exception:

[jcristau]

The agent is calling /api/v2/releases to get all releases, then filtering it to those with scheduled changes. We could have an API endpoint that does the filtering server-side (v1 has /scheduled_changes/releases).

[jcristau]

FWIW I didn't reproduce the 502, but /v2/releases/ is quite slow...

$ time curl -I https://aus4-admin.mozilla.org/api/v2/releases 
HTTP/2 200 
server: nginx
date: Mon, 02 Dec 2024 09:27:05 GMT
content-type: application/json
content-length: 1094240
vary: Accept-Encoding
vary: Accept-Encoding
x-frame-options: DENY
x-frame-options: DENY
x-content-type-options: nosniff
x-content-type-options: nosniff
strict-transport-security: max-age=31536000;
strict-transport-security: max-age=31536000
access-control-allow-headers: Authorization, Content-Type
access-control-allow-methods: OPTIONS, GET, POST, PUT, DELETE
content-security-policy: default-src 'none'; frame-ancestors 'none'
via: 1.1 google
alt-svc: clear


real	0m12.591s
user	0m0.044s
sys	0m0.016s

[jcristau]

As an example of inefficiency, in

balrog/src/auslib/services/releases.py

Line 258 in 3f7ed2e

def get_releases(trans, args=None):

we loop over releases and call getPotentialRequiredSignoffs for each, so that's O(n) sql queries, which is ungood.