Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

obs: export metrics about MemStats into timeseries #96717

Closed
1lann opened this issue Feb 7, 2023 · 0 comments · Fixed by #119819
Closed

obs: export metrics about MemStats into timeseries #96717

1lann opened this issue Feb 7, 2023 · 0 comments · Fixed by #119819
Labels
A-kv-observability A-observability-inf C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@1lann
Copy link
Contributor

1lann commented Feb 7, 2023 •
edited by exalate-issue-sync bot
Loading

CockroachDB currently exports some useful MemStats derived metrics on the /debug/metrics endpoint, however most of these are not exported to timeseries metrics.

It appears only two of them are:

  • sys.go.allocbytes: HeapAlloc
  • sys.go.totalbytes: Sys - HeapReleased

As we intend to rollout GOMEMLIMIT in our deployments on Cockroach Cloud, we'd like better observability in Prometheus for these memory metrics to better monitor the impact GOMEMLIMIT has, and heap memory usage/garbage collection behavior in general. As such, we'd like the memory metrics already being exported to /debug/metrics to be exported to timeseries as well, specifically:

  • MemStackSysBytes: StackSys
  • HeapFragmentBytes: HeapInuse - HeapAlloc
  • HeapReservedBytes: HeapIdle - HeapReleased
  • HeapReleasedBytes: HeapReleased
  • TotalAlloc: this is not exposed in /debug/metrics, but we'd like this metric to monitor the rate of memory allocations.

Somewhat related to #88178

Jira issue: CRDB-24292

Epic CRDB-34227
@1lann 1lann added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-kv-observability T-observability-inf A-observability-inf labels Feb 7, 2023
@lyang24 lyang24 mentioned this issue Jan 22, 2024
Closed
lyang24 added a commit to lyang24/cockroach that referenced this issue Jan 23, 2024
@lyang24
status: export metrics about MemStats into timeseries
250e32f 
This commit exposes 5 metrics into cockroachdb's RuntimeStatSampler timeseries.
The added metrics are MemStackSysBytes, HeapFragmentBytes, HeapReservedBytes,
HeapReleasedBytes, TotalAlloc. These metrics is derived from go MemStats and
aims to improve monitoring of memoery allocations.

Fixes: cockroachdb#96717

Relase note: None
@lyang24 lyang24 mentioned this issue Mar 1, 2024
Merged
lyang24 added a commit to lyang24/cockroach that referenced this issue Mar 14, 2024
@lyang24
status: export metrics about MemStats into timeseries
f86f758 
This commit exposes 5 metrics into cockroachdb's RuntimeStatSampler timeseries.
The added metrics are MemStackSysBytes, HeapFragmentBytes, HeapReservedBytes,
HeapReleasedBytes, TotalAlloc. These metrics are derived from rumtime/metrics.

Fixes: cockroachdb#96717

Relase note: None
lyang24 added a commit to lyang24/cockroach that referenced this issue Mar 15, 2024
@lyang24
status: export metrics about MemStats into timeseries
6602d67 
This commit exposes 5 metrics into cockroachdb's RuntimeStatSampler timeseries.
The added metrics are MemStackSysBytes, HeapFragmentBytes, HeapReservedBytes,
HeapReleasedBytes, TotalAlloc. These metrics are derived from rumtime/metrics.

Fixes: cockroachdb#96717

Relase note: None
craig bot pushed a commit that referenced this issue Mar 15, 2024
@lyang24 @mgartner
@ajstorm @yuzefovich
Merge #119819 #120384 #120493 #120529
ae00dcb 
119819: status: export metrics about MemStats into timeseries r=lyang24 a=lyang24

This commit exposes 5 metrics into cockroachdb's RuntimeStatSampler timeseries. The added metrics are MemStackSysBytes, HeapFragmentBytes, HeapReservedBytes, HeapReleasedBytes, TotalAlloc. These metrics are derived from rumtime/metrics.

Fixes: #96717

Relase note: None

120384: sql: add FORCE_INVERTED_INDEX hint r=DrewKimball a=mgartner

#### sql/parser: add parse-no-verify test command

The `parse-no-verify` test command has been added to data-driven parser
tests. In addition, the `error` test command now asserts that parsing
the statement results in an error.

Release note: None

#### sql/parser: move index hint tests to new file

This is purely a mechanical movement of parser tests with index hints
into a new file.

Release note: None

#### sql/parser: support FORCE_INVERTED_INDEX hint

This commit adds parsing support for the `FORCE_INVERTED_INDEX` hint. The
hint currently has no effect.

Release note: None

#### sql: support FORCE_INVERTED_INDEX hint

Epic: None

Release note (sql change): The `FORCE_INVERTED_INDEX` hint is now
supported. This makes the optimizer prefer a query plan scan over any
inverted index of the hinted table. The query will result in an error if
no such query plan can be generated.


120493: drt: cleanup workload runner scripts r=itsbilal a=ajstorm

Cleanup the workload runner scripts to:

- Add the chaos test to systemd-run
- Fix a few errors in cct_tpcc_drop
- Add kv and tpcc scripts

Release note: none
Epic: none

120529: util/mon: reduce sizes of BytesMonitor and BoundAccount structs r=yuzefovich a=yuzefovich

This PR audits BytesMonitor and BoundAccount structs in order to clean up their creation and reduce their memory footprint. In particular, BytesMonitor goes from 208 bytes to 160 bytes and BoundAccount from 32 bytes to 24 bytes.

See each commit for more details.

Microbenchmarks are [here](https://gist.github.com/yuzefovich/f61ad24da47dacfee92dc06dd214d26f) - the improvement is solely due to stack allocating a slice of children in one of the commits.

Inspired by https://github.com/cockroachlabs/support/issues/2847.

Epic: None

Co-authored-by: lyang24 <[email protected]>
Co-authored-by: Marcus Gartner <[email protected]>
Co-authored-by: Adam Storm <[email protected]>
Co-authored-by: Yahor Yuzefovich <[email protected]>
@craig craig bot closed this as completed in 13066e1 Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-kv-observability A-observability-inf C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
None yet
Milestone
No milestone
1 participant
@1lann