Replace dict with new hashtable for sets datatype #1176

SoftlyRaining · 2024-10-16T01:23:20Z

The new hashtable provides faster lookups and uses less memory than dict.

A TCL test case "SRANDMEMBER with a dict containing long chain" is deleted because it's covered by a hashtable unit test "test_random_entry_with_long_chain", which is already present.

This change also moves some logic from dismissMemory (object.c) to zmadvise_dontneed (zmalloc.c), so the hashtable implementation which needs the dismiss functionality doesn't need to depend on object.c and server.h.

This PR follows #1186.

codecov · 2024-10-16T01:38:06Z

Codecov Report

Attention: Patch coverage is 75.96567% with 56 lines in your changes missing coverage. Please review.

Project coverage is 66.64%. Comparing base (5f7fe9e) to head (1f51a2f).
Report is 10 commits behind head on unstable.

Files with missing lines	Patch %	Lines
src/module.c	0.00%	25 Missing ⚠️
src/debug.c	0.00%	14 Missing ⚠️
src/defrag.c	68.18%	7 Missing ⚠️
src/rdb.c	85.71%	3 Missing ⚠️
src/server.c	0.00%	3 Missing ⚠️
src/db.c	92.85%	2 Missing ⚠️
src/t_set.c	97.29%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##           unstable    #1176      +/-   ##
============================================
- Coverage     70.90%   66.64%   -4.27%     
============================================
  Files           119      119              
  Lines         64631    64693      +62     
============================================
- Hits          45828    43116    -2712     
- Misses        18803    21577    +2774

Files with missing lines	Coverage Δ
src/hashtable.c	`75.88% <100.00%> (+1.67%)`	⬆️
src/lazyfree.c	`86.11% <100.00%> (ø)`
src/object.c	`82.13% <100.00%> (+1.46%)`	⬆️
src/server.h	`100.00% <ø> (ø)`
src/t_zset.c	`95.66% <100.00%> (-0.01%)`	⬇️
src/zmalloc.c	`82.74% <100.00%> (+0.13%)`	⬆️
src/db.c	`89.55% <92.85%> (+0.04%)`	⬆️
src/t_set.c	`97.84% <97.29%> (+0.33%)`	⬆️
src/rdb.c	`76.74% <85.71%> (+0.28%)`	⬆️
src/server.c	`10.44% <0.00%> (-77.02%)`	⬇️
... and 3 more

... and 14 files with indirect coverage changes

tests/unit/type/set.tcl

zuiderkwast

Partial review. I'll look more later.

tests/unit/type/set.tcl

src/hashset.c

zuiderkwast · 2024-10-18T09:45:53Z

Sorry for force-pushing the hashset branch again, to fix a DCO issue.

Can you rebase and force-push again? (I guess it's better than merge when we have a DCO issue following us.)

zuiderkwast

This looks really good. Just a few nits.

src/db.c

src/hashset.c

src/t_set.c

src/t_zset.c

tests/unit/type/set.tcl

SoftlyRaining · 2024-11-25T22:41:52Z

Thanks Viktor for taking care of the rebase - your changes look good to me. :)

zuiderkwast · 2024-11-25T23:09:28Z

You're welcome. :) The main thing I don't like right now is OBJ_ENCODING_HT for dict vs OBJ_ENCODING_HASHTABLE for hashtable. Maybe we should rename the former to OBJ_ENCODING_DICT and the latter to OBJ_ENCODING_HT instead?

SoftlyRaining · 2024-11-26T00:14:56Z

That's a fair point, but OBJECT_ENCODING_HT will be removed completely when all data types have been converted to hashset, so it is only an issue short term, I think. I don't want to touch extra files just to rename it temporarily - I feel like that might make the PR more confusing and harder to review. :-/

zuiderkwast · 2024-12-06T21:09:17Z

Are you updating this? It becomes very hard to rebase it if you wait this long between each rebase.

SoftlyRaining · 2024-12-07T20:40:20Z

I did rebase about a week ago - might have forgotten to push. Ice just finished my oncall shift and rebasing is at the top of my list. 🙂 really looking forward to getting back to this

…

On Fri, Dec 6, 2024, 1:09 PM Viktor Söderqvist ***@***.***> wrote: Are you updating this? It becomes very hard to rebase it if you wait this long between each rebase. — Reply to this email directly, view it on GitHub <#1176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE5RJXUR3JINHGJFX5BHW2L2EIHBFAVCNFSM6AAAAABQAK2CNWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRUGE3TSMRRHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

zuiderkwast · 2024-12-10T20:56:10Z

I rebased it (started from unstable and applied commit with cherry-pick) but then I forgot git cherry-pick --continue after solving the merge conflict. When I pushed, I just pushed unstable, causing the diff to become empty and then the PR closed. Now I can't push to your branch to fix it. 😢

SoftlyRaining · 2024-12-11T00:10:49Z

I got the PR fixed and back up - ready to review and merge! Thanks @zuiderkwast for your work rebasing this again. 😊

zuiderkwast

Nits.

Changes to the merged code after code review include avoiding the cast to void **. (Related to the strict aliasing rule or just nicer code?) We should do the corresponding changes here.

When I rebased, I just made it compile. I didn't update anything else, like hs variable names.

src/db.c

src/debug.c

src/t_set.c

src/t_zset.c

Signed-off-by: Rain Valentine <[email protected]>

SoftlyRaining · 2024-12-13T23:09:47Z

It seems like most of these check failures are also happening in unstable? :(

zuiderkwast · 2024-12-14T04:38:08Z

It seems like most of these check failures are also happening in unstable? :(

Yes, it's not a blocker for this PR. We should fix them in unstable.

tests/unit/type/set.tcl

The variable just means "a hashtable-backed set". It was used for dict and it's not wrong to use it for the new hashtable. Signed-off-by: Viktor Söderqvist <[email protected]>

The new `hashtable` provides faster lookups and uses less memory than `dict`. A TCL test case "SRANDMEMBER with a dict containing long chain" is deleted because it's covered by a hashtable unit test "test_random_entry_with_long_chain", which is already present. This change also moves some logic from dismissMemory (object.c) to zmadvise_dontneed (zmalloc.c), so the hashtable implementation which needs the dismiss functionality doesn't need to depend on object.c and server.h. This PR follows valkey-io#1186. --------- Signed-off-by: Rain Valentine <[email protected]> Signed-off-by: Viktor Söderqvist <[email protected]> Co-authored-by: Viktor Söderqvist <[email protected]>

SoftlyRaining commented Oct 16, 2024

View reviewed changes

tests/unit/type/set.tcl Show resolved Hide resolved

zuiderkwast reviewed Oct 16, 2024

View reviewed changes

tests/unit/type/set.tcl Show resolved Hide resolved

src/hashset.c Outdated Show resolved Hide resolved

src/hashset.c Outdated Show resolved Hide resolved

zuiderkwast force-pushed the hashset branch from 19576b5 to 8fe59b3 Compare October 17, 2024 13:35

SoftlyRaining force-pushed the set-datatype branch from 60f3f70 to add04d0 Compare October 17, 2024 22:00

zuiderkwast force-pushed the hashset branch from 8fe59b3 to 3038293 Compare October 18, 2024 07:22

zuiderkwast linked an issue Oct 22, 2024 that may be closed by this pull request

Use new hashtable for sets #1094

Closed

SoftlyRaining force-pushed the set-datatype branch from add04d0 to e9280d8 Compare October 22, 2024 22:16

zuiderkwast reviewed Oct 24, 2024

View reviewed changes

src/db.c Outdated Show resolved Hide resolved

src/db.c Outdated Show resolved Hide resolved

src/hashset.c Outdated Show resolved Hide resolved

src/t_set.c Outdated Show resolved Hide resolved

src/t_zset.c Outdated Show resolved Hide resolved

tests/unit/type/set.tcl Show resolved Hide resolved

SoftlyRaining force-pushed the set-datatype branch from e14257b to b5f01d1 Compare November 7, 2024 06:30

zuiderkwast force-pushed the hashset branch 9 times, most recently from 63f785b to 8aa2646 Compare November 20, 2024 19:55

zuiderkwast force-pushed the set-datatype branch from b5f01d1 to 3c30172 Compare November 22, 2024 18:54

zuiderkwast force-pushed the hashset branch 6 times, most recently from c957864 to d6da54f Compare December 2, 2024 21:41

zuiderkwast changed the title ~~[hashset feature] Convert SET datatype to use hashset instead of dict~~ [hashtable feature] Convert SET datatype to use hashtable instead of dict Dec 6, 2024

zuiderkwast force-pushed the hashset branch 2 times, most recently from d090f8e to 3559ba8 Compare December 10, 2024 12:55

zuiderkwast changed the base branch from hashset to unstable December 10, 2024 20:34

zuiderkwast closed this Dec 10, 2024

zuiderkwast force-pushed the set-datatype branch from 3c30172 to 3eb8314 Compare December 10, 2024 20:52

SoftlyRaining reopened this Dec 11, 2024

zuiderkwast reviewed Dec 11, 2024

View reviewed changes

zuiderkwast added the run-extra-tests Run extra tests on this PR (Runs all tests from daily except valgrind and RESP) label Dec 11, 2024

SoftlyRaining force-pushed the set-datatype branch 2 times, most recently from 1138fac to ded690a Compare December 13, 2024 05:40

Replace dict with hashtable: SET datatype

22317e7

Signed-off-by: Rain Valentine <[email protected]>

SoftlyRaining force-pushed the set-datatype branch from ded690a to 22317e7 Compare December 13, 2024 05:52

zuiderkwast reviewed Dec 14, 2024

View reviewed changes

tests/unit/type/set.tcl Outdated Show resolved Hide resolved

zuiderkwast changed the title ~~[hashtable feature] Convert SET datatype to use hashtable instead of dict~~ Replace dict with new hashtable for sets datatype Dec 14, 2024

zuiderkwast reviewed Dec 14, 2024

View reviewed changes

tests/unit/type/set.tcl Outdated Show resolved Hide resolved

tests/unit/type/set.tcl Outdated Show resolved Hide resolved

tests/unit/type/set.tcl Outdated Show resolved Hide resolved

tests/unit/type/set.tcl Outdated Show resolved Hide resolved

Revert renamed variable "myhashset" in test case

1f51a2f

The variable just means "a hashtable-backed set". It was used for dict and it's not wrong to use it for the new hashtable. Signed-off-by: Viktor Söderqvist <[email protected]>

zuiderkwast approved these changes Dec 14, 2024

View reviewed changes

zuiderkwast added release-notes This issue should get a line item in the release notes performance labels Dec 14, 2024

zuiderkwast merged commit 88942c8 into valkey-io:unstable Dec 14, 2024
50 of 56 checks passed

SoftlyRaining deleted the set-datatype branch December 17, 2024 19:34

zuiderkwast mentioned this pull request Jan 17, 2025

[Test failure] LATENCY of expire events are correctly collected (unit/latency-monitor) #1580

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace dict with new hashtable for sets datatype #1176

Replace dict with new hashtable for sets datatype #1176

SoftlyRaining commented Oct 16, 2024 •

edited by zuiderkwast

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading

zuiderkwast left a comment

zuiderkwast commented Oct 18, 2024

zuiderkwast left a comment

SoftlyRaining commented Nov 25, 2024

zuiderkwast commented Nov 25, 2024

SoftlyRaining commented Nov 26, 2024

zuiderkwast commented Dec 6, 2024

SoftlyRaining commented Dec 7, 2024 via email

zuiderkwast commented Dec 10, 2024

SoftlyRaining commented Dec 11, 2024

zuiderkwast left a comment

SoftlyRaining commented Dec 13, 2024

zuiderkwast commented Dec 14, 2024

Replace dict with new hashtable for sets datatype #1176

Replace dict with new hashtable for sets datatype #1176

Conversation

SoftlyRaining commented Oct 16, 2024 • edited by zuiderkwast Loading

codecov bot commented Oct 16, 2024 • edited Loading

Codecov Report

zuiderkwast left a comment

Choose a reason for hiding this comment

zuiderkwast commented Oct 18, 2024

zuiderkwast left a comment

Choose a reason for hiding this comment

SoftlyRaining commented Nov 25, 2024

zuiderkwast commented Nov 25, 2024

SoftlyRaining commented Nov 26, 2024

zuiderkwast commented Dec 6, 2024

SoftlyRaining commented Dec 7, 2024 via email

zuiderkwast commented Dec 10, 2024

SoftlyRaining commented Dec 11, 2024

zuiderkwast left a comment

Choose a reason for hiding this comment

SoftlyRaining commented Dec 13, 2024

zuiderkwast commented Dec 14, 2024

SoftlyRaining commented Oct 16, 2024 •

edited by zuiderkwast

Loading

codecov bot commented Oct 16, 2024 •

edited

Loading