Use state file for updating N+ upstreams

[sjberman]

Problem: When splitting the data and control plane, we want to be able to maintain the ability to dynamically update upstreams without a reload if using NGINX Plus. With the current process, we write the upstreams into the nginx conf but don't reload, then call the API to actually do the update. Writing into the conf will trigger a reload from the agent once we split, however.

There were also two bugs:

• if metrics were disabled, the nginx plus client was not initialized, preventing API calls from occuring and instead a reload occurred
• stream upstreams were not updated with the API

Solution: Don't write the upstream servers into the nginx conf anymore when using NGINX Plus. Instead, utilize the state file option that the NGINX Plus API will populate with the upstream servers. This way we can just call the API and don't unintentionally reload by writing servers into the conf.

Also added support for updating stream upstreams, and fixed the initialization bug.

Testing: Verified that the state file is set and populated when using NGINX Plus. With OSS, servers are still written to the config.

Closes #2841

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

Fix an issue where updating upstreams with the NGINX Plus API would not occur if metrics were disabled.

Support updating stream upstreams with the NGINX Plus API instead of reloading NGINX.

Use state files for NGINX Plus upstream servers instead of the NGINX config.

[codecov]

Codecov Report

Attention: Patch coverage is 91.91176% with 11 lines in your changes missing coverage. Please review.

Project coverage is 89.74%. Comparing base (0cbb726) to head (ddc1ebb).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
internal/mode/static/manager.go	0.00%	5 Missing ⚠️
internal/mode/static/handler.go	95.52%	2 Missing and 1 partial ⚠️
internal/mode/static/nginx/runtime/manager.go	88.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2897      +/-   ##
==========================================
+ Coverage   89.71%   89.74%   +0.03%     
==========================================
  Files         109      109              
  Lines       11091    11150      +59     
  Branches       50       50              
==========================================
+ Hits         9950    10007      +57     
- Misses       1082     1083       +1     
- Partials       59       60       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[bjee19]

I thought we want to have the Separate Data and Control Plane work merged into a feature branch, was I mistaken?

[sjberman]

I thought we want to have the Separate Data and Control Plane work merged into a feature branch, was I mistaken?

This is technically independent of that work, it's just something that has to be done before the split is complete. But it's not a part of that work.