forked from cockroachdb/cockroach
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
tsdb: expand mem per worker based on sql pool size
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. This PR introduces a new CLI flag `--max-tsdb-memory` to mirror the functionality of the `--max-sql-memory` flag by accepting bytes or a percentage of system memory. The default is set to be `1%` of system memory or 64 MiB, whichever is greater. This ensures that performance after this PR is equal or better for timeseries queries without eating too far into memory budgets for SQL. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. Resolves cockroachdb#72986 Release note (cli change, ops change): A new CLI flag `--max-tsdb-memory` is now available, that can set the memory budget for timeseries queries when processing requests from the Metrics page in DB Console. Most customers should not need to tweak this setting as the default of 1% of system memory or 64 MiB, whichever is greater, is adequate for most deployments. In the case where a deployment of hundreds of nodes has low per-node memory available (below 8 GiB for instance) it may be necessary to increase this value to `2%` or higher in order to render timeseries graphs for the cluster using the DB Console. Otherwise, the default settings will be adequate for the vast majority of deployments.
- Loading branch information
1 parent
baeba80
commit 97150df
Showing
7 changed files
with
116 additions
and
70 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters