extend Hasura service boot grace period from 0 to 180s

[freemvmt]

Related to this ticket.

This PR introduces a 180s grace period for the Hasura service in an attempt to deal with a boot loop during scale up.

Context

During load testing, even with relatively small numbers of users we immediately hit issues with Hasura.

When the service tries to scale up with a 2nd Task to deal with the load, the new task fails to boot to a healthy state (which the load balancer checks for before directing traffic to it), and AWS then has to deprovision them (usually within 2 minutes or so of boot).

The issue seems to be with the Hasura container itself, since the HasuraProxy container reports the following in the logs for one of these failed task:

{"level":"error","ts":1733444599.5636377,"logger":"http.log.error","msg":"dial tcp 127.0.0.1:8080: connect: connection refused","request":{"remote_ip":"10.0.69.213","remote_port":"35948","proto":"HTTP/1.1","method":"GET","host":"10.0.1.180","uri":"/healthz","headers":{"Connection":["close"],"User-Agent":["ELB-HealthChecker/2.0"],"Accept-Encoding":["gzip, compressed"]}},"duration":0.000346896,"status":502,"err_id":"swhm20qqn","err_trace":"reverseproxy.statusError (reverseproxy.go:1299)"}

Note the 502 Bad Gateway status code, the reverseproxy.statusError (reverseproxy.go:1299) error trace (source), and the very short duration of the request.

See Notion doc for more detail.

[github-actions]

Removed vultr server and associated DNS entries