-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Use WAL Manager #13491
feat: Use WAL Manager #13491
Conversation
This commit changes the RF-1 ingester to use the WAL Manager instead of flushCtx.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, this looks great! I left a few comments but nothing major so I'm happy to approve.
@@ -213,9 +217,18 @@ func (s *stream) storeEntries(ctx context.Context, entries []*logproto.Entry, us | |||
|
|||
bytesAdded += len(entries[i].Line) | |||
} | |||
flushCtx.segmentWriter.Append(s.tenant, s.labels.String(), s.labels, entries) | |||
|
|||
res, err := w.Append(wal.AppendRequest{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It might be more performant to use a pointer for Append here - unsafe.Sizeof
says the AppendRequest is 80B and since it's passed by value it gets copied into the function call.
I'm not sure if this actually leads to any performance benefit in this case but might be worth a benchmark since it's called so often?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of the time &wal.AppendRequest{}
will be turned into wal.AppendRequest{}
at compile-time as it's faster to do this copy on the stack instead of allocate it on the heap. It's easier to show with an isolated example.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Benchmark for using the stack:
go test -gcflags=-N -bench=. bench_test.go
goos: darwin
goarch: arm64
BenchmarkDoRequest-8 573065010 2.098 ns/op
PASS
ok command-line-arguments 2.663s
type AppendRequest struct {
TenantID string
Labels map[string]string
LabelsStr string
}
func doRequest(r AppendRequest) {
_ = r
}
func BenchmarkDoRequest(t *testing.B) {
labels := map[string]string{"foo": "bar"}
labelsStr := "{foo=\"bar\"}"
for i := 0; i < t.N; i++ {
doRequest(AppendRequest{
TenantID: "1",
Labels: labels,
LabelsStr: labelsStr,
})
}
}
Benchmark for using the heap:
go test -gcflags=-N -bench=. bench_test.go
goos: darwin
goarch: arm64
BenchmarkDoRequest-8 346227085 3.131 ns/op
PASS
ok command-line-arguments 1.654s
type AppendRequest struct {
TenantID string
Labels map[string]string
LabelsStr string
}
func doRequest(r *AppendRequest) {
_ = r
}
func BenchmarkDoRequest(t *testing.B) {
labels := map[string]string{"foo": "bar"}
labelsStr := "{foo=\"bar\"}"
for i := 0; i < t.N; i++ {
doRequest(&AppendRequest{
TenantID: "1",
Labels: labels,
LabelsStr: labelsStr,
})
}
}
i.flushQueues[0].Enqueue(currentFlushCtx) | ||
for { | ||
// Keep adding ops to the queue until there are no more. | ||
it, _ := i.wal.NextPending() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we could remove the err return from NextPending if we don't look at it?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think so too, I'll do it for both NextPending
and Put
in another PR. 👍
What this PR does / why we need it:
This commit changes the RF-1 ingester to use the WAL Manager instead of
flushCtx
.Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CONTRIBUTING.md
guide (required)feat
PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.docs/sources/setup/upgrade/_index.md
production/helm/loki/Chart.yaml
and updateproduction/helm/loki/CHANGELOG.md
andproduction/helm/loki/README.md
. Example PRdeprecated-config.yaml
anddeleted-config.yaml
files respectively in thetools/deprecated-config-checker
directory. Example PR