Memory spikes when (re)loading nginx configuration

[dbaumgarten]

Hi,

whenever nginx reload's it's configuration we see quite a large spike in memory-consumption for the pod.
The more reloads happen in a short timeframe the larger that spike is.

I understand that is because of the way such a reload works (new worker processes, draining of existing processes etc).

However I am a little surprised by the dimensions of that increase.

Here is a screenshot of the memory-consumption of nginx pods when they are beeing replaced by other pods via a rolling update.
(The behavior is very similar when just a config-reload is peformed)

As you can see the memory-usage of a pods spkies from <2GB to around ~10GB. A 5-fold increase.

Is such a large increase really normal? Is there something going wrong?

Given that reloads might happen when nginx is under high load (and therefore autoscaling increases the number of nginx and backend pods) that can become an issue.

High Load -> High Resource Usage -> Autoscaling triggers -> New Pods are created -> Config reload because of new pods -> Even higher Resource Usage because of the reloads.

Currently we have solved the issue by simply setting very high memory requests and limits (12GB).
But setting a 12 GB request for a pod that usually uses ~3GB of memory just feels wrong.

[jjngx]

@dbaumgarten could you please provide more information about the platform you use?

[dbaumgarten]

Sure, I can provide all the information you need. Is there anything specific?
We are running our stuff on AWS EKS with managed worker nodes (m6a.2xlarge).
We are running two sets of nginx-ingress-controllers, each with similar settings but responsible for different ingress-classes.
We have autoscaling enabled for nodes and nginx pods, but usually there are 4 nodes and 2 nginx pods per ingress-class.
There are three ingress-objects per ingress-class, each with a bunch of routes.

These ingress-objects aren't super complex, but one thing that is weird is: a kubectl describe on such an ingress-object takes 9 seconds?!

Also: When looking at the metric nginx_ingress_controller_nginx_last_reload_milliseconds it shows that a reload takes over 8 seconds. That seems quite much to me

Here is an example of a typical ingress-object of ours: ingress.txt

[j1m-ryan]

Have you seen any improvement in 3.7? There was a bug fixed in 3.7 where batch reloads weren't being turned off. This was a bug that would also only happen under high changes to trigger batch reloads.

#6446

[dbaumgarten]

We are currently running v3.6.0.
I will test v3.7 on our dev-envionment and come back with results

[dbaumgarten]

While I was testing out v3.7 on our dev environment we had a major outage of our prod environmen.
A regular restart of the nginx-ingress-pods caused a gigantic memory spike (from usually <2GB to ~22GB) which killed the whole node (despite having a memory-limit set) and whenever a pod was scheduled on a new node it promptly killed that one too.

We have for now set limits and requests for the nginx-ingress pods to 25GB, which seems to be enough for them to come up and occasionally reload their config.

We are now (a little) panically trying to figure out what is causing these spikes, as having them request 12x the usual amount of memory is not really a long-term solution.

[vepatel]

@dbaumgarten Can you please give us a rough idea of total apps, VS/VSR or Ingress resources in your cluster and if there are any other resources not relevant to NIC deployment?
Also was the issue started after upgrading to a specific version of NIC?

[dbaumgarten]

Sure,
the number of apps in the cluster is actually pretty low. There are only 2 apps that are handled by the ingresses.
However these apps need to be reachable under different domains (which sometimes require different SSL settings) and some path's need to be handled differently.

Thats why a total of 3 ingress objects per ingress-class, with 4 domains each in the ingress-onject and 6 paths per domain.

Could the fact that so many routes are pointing to the same service cause issues? Would a change to the service's endpoints trigger multiple reloads because the service is used in multiple ingress-objects/routes?

Apart from this all the cluster is relatively small and does not contain a lot of stuff, nothing of which should influence nginx in any way.

The issue has more or less always been present. But as our ingress-objects are getting more and more complex the siutation seems to worsen dramatically.

However I seem to be unable to reproduce the situation on our dev-environment. I see a memory ingrease there too on reload, but it is waaaay less drastic. It seems to be directly related to the load being handled by nginx (and the complexity of the ingress objects)

[vepatel]

@dbaumgarten you mention "per ingress class", so is it safe to assume that there are multiple ICs watching same service and related backend?

[dbaumgarten]

Yes, there are exactly two deployments of nginx-ingress (intranet and internet), each watching their own ingressClass.
Configuration on both deployments is identical. Usualy each ingress-object exists twice (once per ingressClass) with minimal differences (for example one ingress-object might blacklist a specific path):

Both ingress-deployments share the same backend-services. You basically have two ways to reach the backend services, either via intranet-ingress-deployment or internet-ingress-deployment, depending from where (notwork-wise) your requests are coming.

[vepatel]

However I seem to be unable to reproduce the situation on our dev-environment. I see a memory ingrease there too on reload, but it is waaaay less drastic.

speculating here a bit but that could be due to the fix in 3.7.0, can you change the dev version to NIC 3.6.2 and see if situation is still same.

[dbaumgarten]

Dev is also still on v3.6. I wanted to reproduce the behavior of prod on dev before upgrading so I can see if upgrading actually makes a difference

[dbaumgarten]

I have gathered a few screenshots from our monitoring to illustrate the situation that happend duing the outage on friday:

Memory Usage of pods

Reloads per minute

Duration of reloads

HTTP Requests per minute per pod

Worker processes per pod

Workqueue depth per pod

[vepatel]

@dbaumgarten I see sudden increase in incoming requests in the graph above between 9 and 10, with HPA configured it'll try to increase replicas. I'm thinking if this is what caused the node to run out of memory, have you configured max number of replicas for both NIC and backend deployments?

[dbaumgarten]

Between 9 and 10 things were turbulent. Nginx pods restarted and went into crash loop due to OOM. Therefore at times a single pod had to handle all the requests.
Also sometimes no pod was available and as soon as one became available again it had to handle lots of requests that queued up on the Client-Side while no pod was available.

There was practically no auto scaling involved during the outage. The drastic increase in memory-consumption was caused purely by restarting existing pods.

[dbaumgarten]

We tried to remove the memory limit from the nginx pods to get the pods out of the crash loop. However that was a big mistake, as without the memory limit the memory hungry pods brought down entire nodes (the default values for system-reserved resources on AWS EKS seem to be pretty bad...)

[vepatel]

@dbaumgarten Can you please let us know if were able to run a dev upgrade to 3.7.0 and saw any difference?

[dbaumgarten]

I am currently on vacation, but I will continue working on this once I am back.
The main issue currently is that I seem to be unable to reproduce the behavior on dev. I do see memory spikes there, but far less drastic then on prod. Seems to be something with the nature of traffic on prod. I will keep trying and let you know of any results.

[dbaumgarten]

Hi again,

we have decided to simply update prod to v3.7 in the next possible change window and check if that changes anything. I will keep you updated on this.

Meanwhile I continued investigating. I suspected that the long reload-times (measured by the metric nginx_ingress_controller_nginx_last_reload_milliseconds ) might be part of the problem (longer reloads -> more stuff can happen during a reload).
I tiral-and-error'd arround a bit and found something interesting. The use of CRLs seems to be the cause for the long reload-times.

Once I remove the ssl_crl directive from the server-snippets, reload-times drop from ~6 seconds to ~1 second.
However I have no idea why. The CRL-pem is about 18MB large. That seems kinda reasonable to me.

Any idea why the use of ssl_crl causes such a drastic increase in reload-times?

[dbaumgarten]

Hi,
just wanted to report back with some status updates.

The update to v3.7.0 improved the situation and reduced the frequency and size of memory spikes during reload.
However we still hat very large memory-spikes now and then.

Updating to v3.7.1 however was (for whatever reason) a complete gamechanger!
After the update to v3.7.1 memory-usage (during normal operations) dropped by 75(!!!)%.
Also memory spikes on reloads are barely noticable now.
Additionaly the values for the metric nginx_ingress_controller_nginx_last_reload_milliseconds dropped from ~8000 to ~1000.

I have no idea what exactly changed with v3.7.1 compared to v3.7.0, but it drastically improved the whole situation for us.

https://nginx.org/en/CHANGES mentions for v1.27.2: "Feature: SSL certificates, secret keys, and CRLs are now cached on start or during reconfiguration."
I already suspected it to be something with the CRLs. Maybe that feature helped out here?

[j1m-ryan]

Thrilled to see this @dbaumgarten! Yeah it looks like you were right that it was the ssl_crl directive.

[vepatel]

fantastic stuff @dbaumgarten