Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally prepending s3 compatible storage's key with a hash of the … #5495

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

fulmicoton
Copy link
Contributor

…key.

The point is to workaround S3 rate limiting.
Since it is based on keys, our ULID naming scheme can lead to hotspot in the keyspace.

This solution has a downside. External scripts listing files will have a their job multiplied. For this reason, the prefix cardinality is configurable.

Closes #4824

@fulmicoton fulmicoton requested a review from guilload October 17, 2024 06:07
Copy link

github-actions bot commented Oct 17, 2024

On SSD:

Average search latency is 0.986x that of the reference (lower is better).
Ref run id: 3862, ref commit: e08eb9f
Link

On GCS:

Average search latency is 0.999x that of the reference (lower is better).
Ref run id: 3863, ref commit: e08eb9f
Link

@fulmicoton fulmicoton force-pushed the issue/4824-prefix-hash branch 2 times, most recently from 1b0aa58 to d207e72 Compare October 17, 2024 08:40
…key.

The point is to workaround S3 rate limiting.
Since it is based on keys, our ULID naming scheme can lead to hotspot in
the keyspace.

This solution has a downside. External scripts listing files will have a their job multiplied.
For this reason, the prefix cardinality is configurable.

Closes #4824
@fulmicoton fulmicoton force-pushed the issue/4824-prefix-hash branch from d207e72 to 8c90035 Compare October 17, 2024 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add some random prefix to split ids.
1 participant