Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: add cluster setting to expire sticky bits #37728

Merged
merged 1 commit into from
May 24, 2019

Conversation

jeffrey-xiao
Copy link
Contributor

Since we represent sticky bits as HLCs, we can provide a setting to have
them expire after a certain duration. After this duration, these
manually split ranges will be eligible for automatic merging again.

This setting is useful for cases when a user might want to split the
data as it's being loaded into tables, but after the loading is
complete, the user would want the manual splits be merged as determined
by the merge queue.

Resolves: #37696

Release note (general change): Add kv.range_merge.manual_split_ttl setting

@jeffrey-xiao jeffrey-xiao requested a review from a team May 22, 2019 15:16
@cockroach-teamcity
Copy link
Member

This change is Reviewable

@jeffrey-xiao jeffrey-xiao force-pushed the manual-split-ttl branch 2 times, most recently from 57ee310 to e337f57 Compare May 22, 2019 15:50
Copy link
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @jeffrey-xiao, and @nvanbenschoten)


pkg/storage/merge_queue.go, line 65 at r1 (raw file):

// a LHS and a RHS, then the merge queue will not consider to merge the RHS
// with the range to its left until ManualSplitTTL time has passed.
var ManualSplitTTL = settings.RegisterDurationSetting(

RegisterNonNegativeDurationSetting is probably a better choice.


pkg/storage/merge_queue.go, line 67 at r1 (raw file):

var ManualSplitTTL = settings.RegisterDurationSetting(
	"kv.range_merge.manual_split.ttl",
	"if nonzero, manual splits newer than this duration will not be considered for automatic merging.",

The linter will complain about that ., just drop it


pkg/storage/merge_queue.go, line 307 at r1 (raw file):

	if rhsDesc.StickyBit != nil {
		manualSplitTTL := ManualSplitTTL.Get(&mq.store.ClusterSettings().SV)
		manualSplitExpired := manualSplitTTL != 0 && rhsDesc.StickyBit.GoTime().Add(manualSplitTTL).Before(mq.store.Clock().Now().GoTime())

This doesn't use the logical portion of the HLC, no reason to pay for it. How about mq.store.Clock().PhysicalTime()?

Copy link
Contributor Author

@jeffrey-xiao jeffrey-xiao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner and @nvanbenschoten)


pkg/storage/merge_queue.go, line 65 at r1 (raw file):

Previously, ajwerner wrote…

RegisterNonNegativeDurationSetting is probably a better choice.

Done.


pkg/storage/merge_queue.go, line 67 at r1 (raw file):

Previously, ajwerner wrote…

The linter will complain about that ., just drop it

Done.


pkg/storage/merge_queue.go, line 307 at r1 (raw file):

Previously, ajwerner wrote…

This doesn't use the logical portion of the HLC, no reason to pay for it. How about mq.store.Clock().PhysicalTime()?

Done.

Copy link
Member

@nvanbenschoten nvanbenschoten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 3 of 3 files at r1.
Reviewable status: :shipit: complete! 0 of 0 LGTMs obtained (waiting on @ajwerner, @jeffrey-xiao, and @nvanbenschoten)


pkg/storage/client_merge_test.go, line 3288 at r1 (raw file):

	storagebase.MergeQueueEnabled.Override(sv, true)
	storage.MergeQueueInterval.Override(sv, 0) // process greedily
	storage.ManualSplitTTL.Override(sv, time.Hour)

pull time.Hour into a constant so you can use it here and below.


pkg/storage/client_merge_test.go, line 3474 at r1 (raw file):

	})

	t.Run("merge split ttl", func(t *testing.T) {

"sticky bit ttl"?


pkg/storage/merge_queue.go, line 62 at r1 (raw file):

// ManualSplitTTL is the amount of time that the merge queue will not consider
// a manually split range for merging if it is nonzero. If a range is split in

And what if it is zero? Are all manual splits considered for merges or are none?


pkg/storage/merge_queue.go, line 67 at r1 (raw file):

var ManualSplitTTL = settings.RegisterDurationSetting(
	"kv.range_merge.manual_split.ttl",
	"if nonzero, manual splits newer than this duration will not be considered for automatic merging.",

I think it would be clearer if we inverted this phrasing. Something like "if nonzero, manual splits older than this duration will be considered for automatic Range merging."

@jeffrey-xiao jeffrey-xiao force-pushed the manual-split-ttl branch 4 times, most recently from adafe2d to f7e4643 Compare May 23, 2019 16:05
@jeffrey-xiao jeffrey-xiao requested review from nvanbenschoten and ajwerner and removed request for nvanbenschoten May 23, 2019 16:24
Copy link
Member

@nvanbenschoten nvanbenschoten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r2.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (waiting on @ajwerner)

Since we represent sticky bits as HLCs, we can provide a setting to have
them expire after a certain duration. After this duration, these
manually split ranges will be eligible for automatic merging again.

This setting is useful for cases when a user might want to split the
data as it's being loaded into tables, but after the loading is
complete, the user would want the manual splits be merged as determined
by the merge queue.

Resolves: cockroachdb#37696

Release note (general change): Add kv.range_merge.manual_split_ttl
                               setting
Copy link
Contributor

@ajwerner ajwerner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r3.
Reviewable status: :shipit: complete! 1 of 0 LGTMs obtained (and 1 stale) (waiting on @ajwerner)

@jeffrey-xiao
Copy link
Contributor Author

bors r+

craig bot pushed a commit that referenced this pull request May 24, 2019
37728: storage: add cluster setting to expire sticky bits r=jeffrey-xiao a=jeffrey-xiao

Since we represent sticky bits as HLCs, we can provide a setting to have
them expire after a certain duration. After this duration, these
manually split ranges will be eligible for automatic merging again.

This setting is useful for cases when a user might want to split the
data as it's being loaded into tables, but after the loading is
complete, the user would want the manual splits be merged as determined
by the merge queue.

Resolves: #37696

Release note (general change): Add kv.range_merge.manual_split_ttl setting

Co-authored-by: Jeffrey Xiao <[email protected]>
@craig
Copy link
Contributor

craig bot commented May 24, 2019

Build succeeded

@craig craig bot merged commit 07a451c into cockroachdb:master May 24, 2019
jeffrey-xiao added a commit to jeffrey-xiao/cockroach that referenced this pull request Jun 4, 2019
Reverts cockroachdb#37728. A cluster setting is coarse to be useful. There could be
instances where multiple people are working in the same cluster and
there would be conflicts on when they want their sticky bits to expire.

Release note(general change): Remove kv.range_merge.manual_split_ttl
                              setting
jeffrey-xiao added a commit to jeffrey-xiao/cockroach that referenced this pull request Jun 4, 2019
Reverts cockroachdb#37728. A cluster setting is coarse to be useful. There could be
instances where multiple people are working in the same cluster and
there would be conflicts on when they want their sticky bits to expire.

Release note(general change): Remove kv.range_merge.manual_split_ttl
                              setting
@jeffrey-xiao jeffrey-xiao deleted the manual-split-ttl branch June 19, 2019 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

storage: add cluster setting to expire manual split sticky bits
4 participants