Our retry-backoff decorators don't work

[MattWellie]

metamist/metamist/graphql/__init__.py

Lines 145 to 151 in d433e62

	@backoff.on_exception(
	backoff.fibo,
	exception=(HTTPError, JSONDecodeError, TransportServerError),
	max_time=55,
	max_value=55,
	)
	def query(

Related: populationgenomics/production-pipelines#950

During pipeline startup we still frequently run into gql failures, and the backoffs being provided by this backoff library are garbage. To my eye this is configured correctly, but the results are trash:

2024-10-24 01:23:33 INFO (backoff 105): Backing off query(...) for 0.3s (gql.transport.exceptions.TransportServerError: 504 Server Error: Gateway Timeout for url: https://sample-metadata-api-mnrpw3mdza-ts.a.run.app/graphql)
2024-10-24 01:23:33 INFO (root 342): Getting sequencing groups for dataset DS1
2024-10-24 01:23:34 INFO (root 342): Getting sequencing groups for dataset DS2
2024-10-24 01:28:34 INFO (backoff 105): Backing off query(...) for 0.5s (gql.transport.exceptions.TransportServerError: 504 Server Error: Gateway Timeout for url: https://sample-metadata-api-mnrpw3mdza-ts.a.run.app/graphql)
2024-10-24 01:33:35 ERROR (backoff 120): Giving up query(...) after 2 tries (gql.transport.exceptions.TransportServerError: 504 Server Error: Gateway Timeout for url: https://sample-metadata-api-mnrpw3mdza-ts.a.run.app/graphql)

@milo-hyben has implemented decorators in prod-pipes using tenacity that might be better. I've also hand-rolled an alternative here to deal with the even flakier PanelApp server.