Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automate scale test #1926

Merged
merged 21 commits into from
May 17, 2024
Merged

Conversation

pleshakov
Copy link
Contributor

@pleshakov pleshakov commented May 2, 2024

Proposed changes

Problem:
Non-functional scale test needs to be run manually.

Solution:

  • Automate scale test.
  • Use in-cluster Prometheus to collect CPU, memory and NGF metrics.
  • Use Kubernetes API server to get NGF logs.

For development and troubleshooting, it is possible to run scale test locally in Kind cluster. However, it is necessary to bring down the number of HTTPRoutes to 50 or less (roughly).

Testing:

  • Ran this test locally with 64 listeners, 50 routes and 50 upstreams with NGINX OSS.
  • Ran this test locally with 64 listeners, 10 routes and 10 upstreams with NGINX Plus.
  • Ran this test on GKE with the default configuration with NGINX OSS. Test results -- pleshakov@55be967

Out of scope: ensuring this test runs successfully via GitHub pipeline -- #1927

Closes #1368

Largely based on work by Ciara in
#1804

Checklist

Before creating a PR, run through this checklist and mark each as complete.

  • I have read the CONTRIBUTING doc
  • I have added tests that prove my fix is effective or that my feature works
  • I have checked that all unit tests pass after adding my changes
  • I have updated necessary documentation
  • I have rebased my branch onto main
  • I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

Release notes

If this PR introduces a change that affects users and needs to be mentioned in the release notes,
please add a brief note that summarizes the change.

NONE

@pleshakov pleshakov requested a review from a team as a code owner May 2, 2024 19:57
@github-actions github-actions bot added the tests Pull requests that update tests label May 2, 2024
tests/framework/request.go Fixed Show fixed Hide fixed
tests/framework/request.go Outdated Show resolved Hide resolved
tests/framework/resourcemanager.go Show resolved Hide resolved
tests/framework/results.go Outdated Show resolved Hide resolved
tests/framework/results.go Outdated Show resolved Hide resolved
tests/suite/scale_test.go Outdated Show resolved Hide resolved
tests/suite/scale_test.go Outdated Show resolved Hide resolved
tests/suite/scale_test.go Outdated Show resolved Hide resolved
tests/suite/scale_test.go Outdated Show resolved Hide resolved
tests/framework/prometheus.go Outdated Show resolved Hide resolved
tests/framework/prometheus.go Show resolved Hide resolved
@sjberman sjberman mentioned this pull request May 2, 2024
6 tasks
Copy link
Contributor

@kate-osborn kate-osborn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Just some small changes

tests/framework/prometheus.go Outdated Show resolved Hide resolved
tests/framework/prometheus.go Outdated Show resolved Hide resolved
tests/framework/resourcemanager.go Outdated Show resolved Hide resolved
tests/framework/resourcemanager.go Outdated Show resolved Hide resolved
tests/suite/scale_test.go Show resolved Hide resolved
@pleshakov pleshakov force-pushed the tests/automate-scale-test branch from 3d61535 to e216554 Compare May 16, 2024 14:50
@pleshakov pleshakov requested review from kate-osborn and sjberman May 16, 2024 20:04
tests/suite/scale_test.go Outdated Show resolved Hide resolved
@sjberman
Copy link
Contributor

If you wanted to verify that this test can be run in the pipeline and close out #1927, you could create this branch on the upstream repo and then run the workflow on that branch.

@pleshakov
Copy link
Contributor Author

If you wanted to verify that this test can be run in the pipeline and close out #1927, you could create this branch on the upstream repo and then run the workflow on that branch.

@sjberman it needs a bigger cluster, that's why I'd rather not update it here

Copy link
Contributor

@sjberman sjberman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One last thing...can we move the scale results and delete the scale folder?

Copy link
Contributor

@kate-osborn kate-osborn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

pleshakov and others added 12 commits May 17, 2024 15:47
Problem:
Non-functional scale test needs to be run manually.

Solution:
- Automate scale test.
- Use in-cluster Prometheus to collect CPU, memory and NGF metrics.
- Use Kubernetes API server to get NGF logs.

For development and troubleshooting, it is possible to run scale test
locally in Kind cluster. However, it is necessary to bring down
the number of HTTPRoutes to 50 or less (roughly).

Testing:
- Ran this test locally with 64 listeners, 50 routes and 50 upstreams
  with NGINX OSS.
- Ran this test on GKE with the default configuration with NGINX OSS.

Out of scope: ensuring this test runs successfully via GitHub pipeline.

Closes nginxinc#1368

Largely based on work by Ciara in
nginxinc#1804

Co-authored-by: Ciara Stacke <[email protected]>
@pleshakov pleshakov force-pushed the tests/automate-scale-test branch from 83fc7bd to 0a4680e Compare May 17, 2024 19:55
@github-actions github-actions bot added the documentation Improvements or additions to documentation label May 17, 2024
@pleshakov pleshakov merged commit ee4319c into nginxinc:main May 17, 2024
40 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation tests Pull requests that update tests
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

Automate Scale Test
4 participants