504 in the frontend facia app

[alinaboghiu]

Tasks

Preview Give feedback

1. Try reducing the number of instances that render fronts (from 27 to 18) #10396
   +0
   
2. Refactor DCR CDK scaling configuration to allow variations between apps #10408
   +0
   
3. Create Fronts DCR App #9323
   +0
   
4. Try running the new fronts rendering app on different sized instances

[cemms1]

The volume of ELB 5xx on the rendering app drops significantly when directing fronts-based traffic to the new facia-rendering app

Graph of rendering app showing the requests vs errors and latency charts for the period before, during and after changing the app doing the fronts rendering:

[cemms1]

When we change from an ELB to an ALB, load balancer 504 responses turn into 502 responses

HTTP 504 errors on the facia app for DCR requests:

HTTP 502 errors on the facia app for DCR requests:

[cemms1]

Current theory is the following from this AWS troubleshooting page:

The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target

The load balancer receives a request and forwards it to the target. The target receives the request and starts to process it, but closes the connection to the load balancer too early. > This usually occurs when the duration of the keep-alive timeout for the target is shorter than the idle timeout value of the load balancer. Make sure that the duration of the keep-alive timeout is greater than the idle timeout value.

Check the values for the request_processing_time, target_processing_time and response_processing_time fields.

See the following example access log entry:

http 2022-04-15T16:52:50.757968Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.001 4.205 -1 502 - 94 326 "GET http://example.com:80 HTTP/1.1" "curl/7.51.0" - - arn:aws:elasticloadbalancing:us-east-1:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337262-36d228ad5d99923122bbe354"

Note: In this access log entry, the request_processing_time is 0.001, the target_processing_time is 4.205, and the response_processing_time is -1.

Useful links:

• Troubleshooting HTTP 502 on application load balancers
• Troubleshooting HTTP 504 on classic load balancers
• Connection idle timeout docs on application load balancers

[cemms1]

I am fairly sure the 502s, and previously 504s, were being caused by the target server keep alive timeout being shorter than the load balancer connection idle timeout

The options here are:

• increase the keep alive timeout on the node application
  e.g. in server.prod.ts:

  const server = app.listen(port);
  server.keepAliveTimeout = 90 * 1000; // ensure this is higher than the default LB idle timeout of 60 seconds
  

• decrease the idle timeout on the load balancer
  e.g. in the screenshot of the LB settings below, ensure the timeout is lower than the Node default timeout of 5 seconds

  

There's a blog post about this issue here, which is worth a quick read

[alinaboghiu]

This write up is fantastic, thank you Charlotte, brilliant 🕵️ work.

[cemms1]

With guidance from DevX, decided to go with option 2 to decrease the idle timeout on the load balancer. The PR is here