ClientSettingsPolicy: proxy_read_timeout

[AlexEndris]

Hi,
just wanted to ask, whether or not it is planned to expose the "proxy_read_timeout" setting through the ClientSettingsPolicy. I'm facing the issue, that some requests take some time to process and the default 60s timeout isn't sufficient, but I don't want to increase this setting globally by using an extraVolume that is then included in the config.

Thanks!

[mpstefan]

It's definitely not fun to configure NGINX that way, which is why we plan to include NGINX snippets as part of our next release, which should allow for this kind of configuration of the NGINX conf.

That said, what kind of timeout are you expecting from proxy_read_timeout? We did a deep dive recently and it turns out this timeout is measuring not individual HTTP transactions, but packets coming from your upstream - meaning if one request takes a long time, but the others are coming back quickly, the one long request will never timeout.

Based on what you've written above, I assume you're looking for a timeout per HTTP transaction?

[AlexEndris]

How would these snippets look like. It sounds intriguing. One main issue I have using extra volumes to configure nginx is, that sometimes, for no apparent reasons, when k8s decides to restart the pod, it sort of forgets to attach the volume properly. Restart of the deployment doesn't fix it, but deletion of the pod does. I've not managed to find out why and how this happens, though, as it's not reproducible by just restarting the deployment etc. So a possibility to configure nginx in a different way (maybe even on route level, as the request I'm talking about is only one particular route) would be very appreciated!

As to your question: Yes, you're right. I've tested it with proxy_read_timeout to work as it should (some requests can take up to 5 or 6 minutes, unfortunately). If you have a better suggestion, I'm all ears.

Btw. I really love how well the gateway works, other than that. Good job :).

[mpstefan]

You can get an idea of how these will work from our enhancement proposal written up in our repository: https://github.com/nginxinc/nginx-gateway-fabric/blob/main/docs/proposals/advanced-nginx-extensions.md

Before these are published, we do have a review period as well if you're interested in any other upcoming features. The other enhancement proposals are available in the directory as well!

@sjberman has the workaround for you below on fixing your issue today.

In the future, we'll need to create our own NJS module for a real HTTP timeout. That timeout will be available within the route, as the Gateway API has already defined timeouts for us, we just need to implement it. I have a stub here that might move up in priority: #2164

Glad to hear you're digging the project! Anything in particular you like about it? Have you tried any the competition (Kong, Envoy Gateway, Istio)?

[AlexEndris]

Thank you, that really sounds nice. Both the snippets and the option to set a proper timeout from the httproute itself. That makes it really much more convenient for the devs here.
No I've not tried the competition. This came as a recommendation from a colleague.

[sjberman]

@AlexEndris Did you happen to view this blog when attempting to patch the nginx config? This process should be a valid workaround for setting the directives you want. Though they will be set globally to the nginx instance.

[sjberman]

The issue is that the NGF control plane fully controls what exists in the conf.d folder, so you can't put your own configuration files in there. It will overwrite anything in there with the files that it builds based on the state of k8s Gateway API configs.

[sjberman]

This is why the workaround above edits the base nginx config file, because the NGF control plane will not overwrite that. The caveat is that it will apply globally.

[AlexEndris]

Interesting. For me it persists quite a long time before it's gone. I didn't know that. I guess, I'll have to add a kustomize patch after the helm chart then, for that. (putting something in the conf.d folder is also global same caveat)

[AlexEndris]

@sjberman Thank you for your suggestion with overriding the base nginx.conf. Considering the ArgoCD approach using the helm chart, it was a hassle to get this going with kustomize, until I noticed, that I've been doing that same workaround the whole time, just a little bit differently. I just put a file into conf.d before, but now I do the same, I mount a volume, as is detailed in the blog post you shared, with the nginx.conf as an extraVolume. It's still a workaround, but it works like a charm until the snippets are done. Thank you!

[sjberman]

Glad to hear it!