ui: graphs don't load on 30 node cluster #24018
Assignees
Comments
|
In addition to this, the logs are full of: |
|
The memory limit on time series queries is set very low at a hard coded level; 64 MiB for the entire system, and 1MiB per query. It looks like 1MiB is not enough to query even the minimum number of keys (3) into memory at one time on a 30 node cluster - we need 43.3K per node, and at 30 nodes we only have 33.3k in the budget. @petermattis Need to make a call here, options are:
Independently of this, I will also fix it so that single-node queries work correctly. |
|
Let's reduce the number of simultaneous workers - I'd go with 8 workers for an 8MiB limit each. |
mrtracy
pushed a commit
to mrtracy/cockroach
that referenced
this issue
Mar 19, 2018
Adjusts time series QueryWorkerMax, the maximum number of simultaneous workers, down to 8 from 64. Also adjusts the estimatedSourceCount for individual queries to match the source count for queries which provide a specific list of sources. Resolves cockroachdb#24018 Release note (Admin UI): Fixed a bug where graphs would not display on clusters with large numbers of nodes.
|
Please backport this! If not for 2.0 then 2.0.1. |
mrtracy
pushed a commit
to mrtracy/cockroach
that referenced
this issue
Mar 19, 2018
Adjusts time series QueryWorkerMax, the maximum number of simultaneous workers, down to 8 from 64. Also adjusts the estimatedSourceCount for individual queries to match the source count for queries which provide a specific list of sources. Resolves cockroachdb#24018 Release note (Admin UI): Fixed a bug where graphs would not display on clusters with large numbers of nodes.
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Jan 11, 2022
Previously, the memory limit per-`tsdb` worker was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated per worker. TODO(davidh): Can the tests be faster? They iterate on a server create TODO(davidh): Is 1/128 a good setting? How do we validate this. TODO(davidh): Should this behavior be gated behind a feature flag? It's possible on some memory-constrained deployments a sudden spike in memory usage by tsdb could cause problems. Resolves cockroachdb#72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics.
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Jan 14, 2022
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated for all running `tsdb` queries. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. This behavior is gated behind a private cluster setting that's enabled by default. TODO(davidh): Can the tests be faster? They iterate on a server create TODO(davidh): Is 1/128 a good setting? How do we validate this. Resolves cockroachdb#72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics.
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Jan 27, 2022
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated for all running `tsdb` queries. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. This behavior is gated behind a private cluster setting that's enabled by default and sets the ratio at 1/128 of the SQL memory pool. Resolves cockroachdb#72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics.
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Jan 31, 2022
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated for all running `tsdb` queries. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. This behavior is gated behind a private cluster setting that's enabled by default and sets the ratio at 1/128 of the SQL memory pool. Resolves cockroachdb#72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics.
dhartunian
added a commit
to dhartunian/cockroach
that referenced
this issue
Feb 2, 2022
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. This PR introduces a new CLI flag `--max-tsdb-memory` to mirror the functionality of the `--max-sql-memory` flag by accepting bytes or a percentage of system memory. The default is set to be `1%` of system memory or 64 MiB, whichever is greater. This ensures that performance after this PR is equal or better for timeseries queries without eating too far into memory budgets for SQL. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. Resolves cockroachdb#72986 Release note (cli change, ops change): A new CLI flag `--max-tsdb-memory` is now available, that can set the memory budget for timeseries queries when processing requests from the Metrics page in DB Console. Most customers should not need to tweak this setting as the default of 1% of system memory or 64 MiB, whichever is greater, is adequate for most deployments. In the case where a deployment of hundreds of nodes has low per-node memory available (below 8 GiB for instance) it may be necessary to increase this value to `2%` or higher in order to render timeseries graphs for the cluster using the DB Console. Otherwise, the default settings will be adequate for the vast majority of deployments.
craig bot
pushed a commit
that referenced
this issue
Feb 15, 2022
74563: kv,kvcoord,sql: poison txnCoordSender after a retryable error r=lidorcarmel a=lidorcarmel Previously kv users could lose parts of a transaction without getting an error. After Send() returned a retryable error the state of txn got reset which made it usable again. If the caller ignored the error they could continue applying more operations without realizing the first part of the transaction was discarded. See more details in the issue (#22615). The simple case example is where the retryable closure of DB.Txn() returns nil instead of returning the retryable error back to the retry loop - in this case the retry loop declares success without realizing we lost the first part of the transaction (all the operations before the retryable error). This PR leaves the txn in a "poisoned" state after encountering an error, so that all future operations fail fast. The caller is therefore expected to reset the txn handle back to a usable state intentionally, by calling Txn.PrepareForRetry(). In the simple case of DB.Txn() the retry loop will reset the handle and run the retry even if the callback returned nil. Closes #22615 Release note: None 74662: tsdb: expand mem per worker based on sql pool size r=dhartunian a=dhartunian Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in #24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. The per-worker limit is now raised based on the amount of memory available to the SQL Pool via the `MemoryPoolSize` configuration variable. This is set to be 25% of the system memory by default. The `tsdb` memory cap per-worker is now doubled until it reaches `1/128` of the memory pool setting. For example, on a node with 128 - 256 GiB of memory, this will correspond to 512 MiB allocated for all running `tsdb` queries. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. This behavior is gated behind a private cluster setting that's enabled by default and sets the ratio at 1/128 of the SQL memory pool. Resolves #72986 Release note (ops change): customers running clusters with 240 nodes or more can effectively access tsdb metrics. 75677: randgen: add PopulateRandTable r=mgartner a=msbutler PopulateRandTable populates the caller's table with random data. This helper function aims to make it easier for engineers to develop randomized tests that leverage randgen / sqlsmith. Informs #72345 Release note: None 76334: opt: fix missing filters after join reordering r=mgartner a=mgartner #### opt: add TES, SES, and rules to reorderjoins This commit updates the output of the `reorderjoins` opt test command to display the initial state of the `JoinOrderBuilder`. It adds additional information to the output including the TES, SES, and conflict rules for each edge. Release note: None #### opt: fix missing filters after join reordering This commit eliminates logic in the `assoc`, `leftAsscom`, and `rightAsscom` functions in the join order builder that aimed to prevent generating "orphaned" predicates, where one or more referenced relations are not in a join's input. In rare cases, this logic had the side effect of creating invalid conflict rules for edges, which could prevent valid predicates from being added to reordered join trees. It is safe to remove these conditionals because they are unnecessary. The CD-C algorithm already prevents generation of orphaned predicates by checking that the total eligibility set (TES) is a subset of a join's input vertices. In our implementation, this is handled by the `checkNonInnerJoin` and `checkInnerJoin` functions. Fixes #76522 Release note (bug fix): A bug has been fixed which caused the query optimizer to omit join filters in rare cases when reordering joins, which could result in incorrect query results. This bug was present since v20.2. Co-authored-by: Lidor Carmel <[email protected]> Co-authored-by: David Hartunian <[email protected]> Co-authored-by: Michael Butler <[email protected]> Co-authored-by: Marcus Gartner <[email protected]>
RajivTS
pushed a commit
to RajivTS/cockroach
that referenced
this issue
Mar 6, 2022
Previously, the memory limit for all `tsdb` workers was set at a static 64MiB. This cap created issues seen in cockroachdb#24018 where this limit was hit on a 30 node cluster. To alleviate the issue, the number of workers was reduced, raising the per-worker allocation. We've currently hit this limit again as part of load testing with larger clusters and have decided to make the per-query worker memory limit dynamic. This PR introduces a new CLI flag `--max-tsdb-memory` to mirror the functionality of the `--max-sql-memory` flag by accepting bytes or a percentage of system memory. The default is set to be `1%` of system memory or 64 MiB, whichever is greater. This ensures that performance after this PR is equal or better for timeseries queries without eating too far into memory budgets for SQL. In addition, the ts server is now connected to the same `BytesMonitor` instance as the SQL memory monitor and workers will becapped at double the query limit. Results are monitored as before but a cap is not introduced there since we didn't have one present previously. Resolves cockroachdb#72986 Release note (cli change, ops change): A new CLI flag `--max-tsdb-memory` is now available, that can set the memory budget for timeseries queries when processing requests from the Metrics page in DB Console. Most customers should not need to tweak this setting as the default of 1% of system memory or 64 MiB, whichever is greater, is adequate for most deployments. In the case where a deployment of hundreds of nodes has low per-node memory available (below 8 GiB for instance) it may be necessary to increase this value to `2%` or higher in order to render timeseries graphs for the cluster using the DB Console. Otherwise, the default settings will be adequate for the vast majority of deployments.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
On a 30 node cluster, graphs hang forever in the Admin UI:
This happens even if I change from "All Nodes" to a single node.
The cause is that the
queryendpoint returns a 500, with the message:The input to that endpoint is:
or
Is there any solution to this besides upping the memory on the boxes? I would hope there's a way to issue whatever queries are necessary on the backend in a less memory-intensive way.
The text was updated successfully, but these errors were encountered: