Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Benchmarking #87

Open
rivernews opened this issue Jun 16, 2020 · 1 comment
Open

Benchmarking #87

rivernews opened this issue Jun 16, 2020 · 1 comment

Comments

@rivernews
Copy link
Owner

rivernews commented Jun 16, 2020 •
edited
Loading

60 parallel scrapers

Memory usage around 5 ~ 6.1G / 8.43G. Seems like we can raise higher to 63 parallel.
image

63 parallel scrapers

Memory usage - SLK around 3.24G at full speed 63 scrapers. Total memory pressure about 5G at startup, probably will fluctuate later on as previous benchmark, still expected to be topped at 6G.
Redis connection amount around 495-505, sometimes can see requests got kicked out by GOAWAY, not sure which part is the bottleneck, Redis, network controller in k8s, or DO.
image
@rivernews
Copy link
Owner Author

rivernews commented Jul 23, 2020 •
edited
Loading

Per-node openj9

Switching to openj9, no particular memory reuse reduction though.
image

Overall fluctuating topped at 2.8G
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rivernews