Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sort namespaces, mounts, and clients before adding them to HLL #28062

Merged
merged 6 commits into from
Aug 13, 2024

Conversation

miagilepner
Copy link
Contributor

Description

Sort the activity log clients before adding them to the hyperloglog to ensure consistent results.

Performance:

$ VAULT_TEST_LOG_DIR=$(mktemp -d) go test -bench=^BenchmarkLargeClientCount$  -tags="ent enterprise testonly" $(pwd)/vault/external_tests/activity_testonly -run "^$" -benchmem 
goos: darwin
goarch: arm64
pkg: github.com/hashicorp/vault/vault/external_tests/activity_testonly
BenchmarkLargeClientCount/10000_per_mount-10                  18          56399891 ns/op        32806902 B/op      50671 allocs/op
BenchmarkLargeClientCount/100000_per_mount-10                  2         762893583 ns/op        207117244 B/op    465944 allocs/op
BenchmarkLargeClientCount/500000_per_mount-10                  1        5710724250 ns/op        1496278752 B/op  2420449 allocs/op
BenchmarkLargeClientCount/1000000_per_mount-10                 1        17837094792 ns/op       2983561024 B/op  4837697 allocs/op
BenchmarkLargeClientCount/2000000_per_mount-10                 1        38098365375 ns/op       5959903936 B/op  9690766 allocs/op
PASS
ok      github.com/hashicorp/vault/vault/external_tests/activity_testonly       145.191s

what's currently on main:

$ VAULT_TEST_LOG_DIR=$(mktemp -d) go test -bench=^BenchmarkLargeClientCount$  -tags="ent enterprise testonly" $(pwd)/vault/external_tests/activity_testonly -run "^$" -benchmem 
goos: darwin
goarch: arm64
pkg: github.com/hashicorp/vault/vault/external_tests/activity_testonly
BenchmarkLargeClientCount/10000_per_mount-10                  22          55692903 ns/op        32134565 B/op      50655 allocs/op
BenchmarkLargeClientCount/100000_per_mount-10                  2         781777021 ns/op        201372028 B/op    466613 allocs/op
BenchmarkLargeClientCount/500000_per_mount-10                  1        5563718292 ns/op        1467175480 B/op  2418694 allocs/op
BenchmarkLargeClientCount/1000000_per_mount-10                 1        11221255292 ns/op       2925794392 B/op  4835923 allocs/op
BenchmarkLargeClientCount/2000000_per_mount-10                 1        23813339459 ns/op       5842432544 B/op  9666556 allocs/op
PASS
ok      github.com/hashicorp/vault/vault/external_tests/activity_testonly       125.793s

Ent PR: https://github.com/hashicorp/vault-enterprise/pull/6369

TODO only if you're a HashiCorp employee

  • Labels: If this PR is the CE portion of an ENT change, and that ENT change is
    getting backported to N-2, use the new style backport/ent/x.x.x+ent labels
    instead of the old style backport/x.x.x labels.
  • Labels: If this PR is a CE only change, it can only be backported to N, so use
    the normal backport/x.x.x label (there should be only 1).
  • ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
    of a public function, even if that change is in a CE file, double check that
    applying the patch for this PR to the ENT repo and running tests doesn't
    break any tests. Sometimes ENT only tests rely on public functions in CE
    files.
  • Jira: If this change has an associated Jira, it's referenced either
    in the PR description, commit message, or branch name.
  • RFC: If this change has an associated RFC, please link it in the description.
  • ENT PR: If this change has an associated ENT PR, please link it in the
    description. Also, make sure the changelog is in this PR, not in your ENT PR.

@miagilepner miagilepner added this to the 1.18.0-rc milestone Aug 13, 2024
@github-actions github-actions bot added the hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed label Aug 13, 2024
Copy link

github-actions bot commented Aug 13, 2024

CI Results:
All Go tests succeeded! ✅

Copy link

github-actions bot commented Aug 13, 2024

Build Results:
All builds succeeded! ✅

Copy link
Contributor

@VioletHynes VioletHynes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Small comment on the changelog but non-blocking

@@ -0,0 +1,3 @@
```release-note:change
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be improvement? change is usually reserved for breaking changes or things that might affect an existing workflow.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, updated

@miagilepner miagilepner merged commit 8cd4263 into main Aug 13, 2024
82 of 83 checks passed
@miagilepner miagilepner deleted the miagilepner/sort-before-hll-insert branch August 13, 2024 14:54
miagilepner added a commit that referenced this pull request Aug 15, 2024
* sort namespaces and mounts before adding them

* also sort clients

* add comments

* pr comment fixes

* changelog

* changelog update
miagilepner added a commit that referenced this pull request Aug 15, 2024
* Handle Query Updates (#27547)

* Simplify handle query to return updated estimation values

* Added changelog

* remove comment

* Revert "Simplify handle query to return updated estimation values"

This reverts commit b67969c.

* temp

* Revert "temp"

This reverts commit 4932979.

* CE files handle query update

* Revert "CE files handle query update"

This reverts commit 8dafa2d.

* CE Changes

* Delete vault/external_tests/upgrade_testing/upgrade_testing_binary/upgrade_test.go

* Sort namespaces, mounts, and clients before adding them to HLL (#28062)

* sort namespaces and mounts before adding them

* also sort clients

* add comments

* pr comment fixes

* changelog

* changelog update

* cleanup after handlequery changes (#27776)

* Show Current Month When Future Month Start Date is Provided (#28042)

* Show current month data

* Added changelog

* Edited changelog

* Client Count Docs Updates/Cleanup (#27862)

* Docs changes

* More condensation of docs

* Added some clarity on date ranges

* Edited wording'

* Added estimation client count info

* Update website/content/api-docs/system/internal-counters.mdx

Co-authored-by: miagilepner <[email protected]>

---------

Co-authored-by: miagilepner <[email protected]>

---------

Co-authored-by: divyaac <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hashicorp-contributed-pr If the PR is HashiCorp (i.e. not-community) contributed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants