feat: Allows to configure client_max_body_size

[satyamsundaram]

What this PR does / why we need it:

Adjusting client_max_body_size in Nginx allows us to fix this nginx error: client intended to send too large body.

Which issue(s) this PR fixes:
Fixes #8736, #10521

Special notes for your reviewer:

Checklist

• Reviewed the CONTRIBUTING.md guide (required)
• Documentation added
• Tests updated
• Title matches the required conventional commits format, see here
  • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
• Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
• For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
• If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

[CLAassistant]

All committers have signed the CLA.

[trevorwhitney]

LGTM, need to bump the version but otherwise let's get this merged!

[satyamsundaram]

I've bumped the version. Let me know if there are any other changes required.

[ursuciprian]

Are there any updates on the merge for this one?

[satyamsundaram]

I've fixed all merge conflicts. It can be merged now.

[jarmd]

@trevorwhitney
Can you rerun the pipeline and potentially merge this ?
This is currently a blocker for us to upgrade to loki 3.x

[morganrowse]

Hi this merge appears to be on the helm Changelog currently on main branch for the Helm 6.15.0 (release 6.18.0 is the latest current release). https://github.com/grafana/loki/blob/f5b0fb6b998dc0a49cd36c0968862340c7e517bf/production/helm/loki/CHANGELOG.md

However I'm not actually finding any chart releases >=6.15.0 that include this variable for consumption.

What is the process here for changes to chart or has this change erroneously been removed or are the changelogs incorrect?

[trevorwhitney]

yeah, it looks like the hem-ci is broken during the installation of the prometheus operator, which is breaking all helm releases. help in fixing it would be greatly appreciated!

[morganrowse]

Thank you, we appreciate the feedback @trevorwhitney 😃

Is there an open issue related to this to track/contribute to?

[trevorwhitney]

@morganrowse there is now 😄 #14799

[asaf400]

I'm still targeting Loki 2.X, so on Chart v5.44.4 which still has the config duplication issue without parameters,
I had to implement a workaround, when supplying chart values, basically inventing a new config value and using it:

gateway:
  nginxConfig:
    # Hack to allow for custom nginx config
    file: | 
      {{- tpl .Values.gateway.nginxConfig.custom_nginx_configfile_template . | indent 2 }}
    
    # Hack to allow for custom nginx config
    # Source from: https://github.com/grafana/loki/blob/helm-loki-5.44.4/production/helm/loki/templates/_helpers.tpl#L648
    custom_nginx_configfile_template: |

      # DO NOT REMOVE EMPTY NEWLINE ABOVE THIS LINE, IT WONT SHOW UP IN FINAL CONFIGMAP BUT IS REQUIRED FOR TEMPLATING
      worker_processes  5;  ## Default: 1
      <.. Rest of config here, with ability for user to modify..>

The upside is that I'm still using official chart source, and no chart modification is required, just values..