feat(storage/dataflux): increase test coverage #11093
Conversation
… all workers as it can nott be reached
| func TestNextBatchContextCancelEmulated(t *testing.T) { | ||
| func TestNextBatchContextCancelEmulated(t *testing.T) { |
There was a problem hiding this comment.
| func TestNextBatchContextCancelEmulated(t *testing.T) { | |
| func TestNextBatchContextCancelEmulated(t *testing.T) { | |
| func TestNextBatchContextCancelEmulated(t *testing.T) { |
| func createObject(ctx context.Context, bucket *storage.BucketHandle, numObjects int, prefix string) error { | ||
| func createObject(ctx context.Context, bucket *storage.BucketHandle, numObjects int, prefix string) error { |
There was a problem hiding this comment.
| func createObject(ctx context.Context, bucket *storage.BucketHandle, numObjects int, prefix string) error { | |
| func createObject(ctx context.Context, bucket *storage.BucketHandle, numObjects int, prefix string) error { | |
| func createObjects(ctx context.Context, bucket *storage.BucketHandle, numObjects int, prefix string) error { |
| if df.method != worksteal { | ||
| t.Errorf("expected df.method to be %v, got %v", worksteal, df.method) | ||
| } |
There was a problem hiding this comment.
How sure are we that this won't be flaky and occasionally sequential wins out?
There was a problem hiding this comment.
Both sequential and worksteal uses gcs api to list objects. Worksteal isn't flaky.
If sequential wins over worksteal, its an error and needs to be caught. For 300k+ objects worksteal should finish faster than sequential listing.
| childCtx, cancel := context.WithCancel(ctx) | ||
| cancel() | ||
| result, err := c.NextBatch(childCtx) | ||
| if err != nil && !strings.Contains(err.Error(), context.Canceled.Error()) { |
There was a problem hiding this comment.
This error check should use errors.Is
There was a problem hiding this comment.
GRPC has context.Canceled.Error() in desc and not as error. So this succeeds for http but fails for GRPC.
| t.Errorf("NextBatch() expected to fail with %v, got nil", context.Canceled) | ||
| } | ||
| if len(result) > 0 { | ||
| t.Errorf("NextBatch() got object %v, want 0 objects", len(result)) |
There was a problem hiding this comment.
nit
| t.Errorf("NextBatch() got object %v, want 0 objects", len(result)) | |
| t.Errorf("NextBatch() got %v objects, want 0 objects", len(result)) |
| // os.Setenv("STORAGE_EMULATOR_HOST", "http://localhost:7000") | ||
| // os.Setenv("STORAGE_EMULATOR_HOST_GRPC", "localhost:8888") |
| desc string | ||
| skipDirectoryObjects bool | ||
| query storage.Query | ||
| method listingMethod |
There was a problem hiding this comment.
I don't see method actually being used.
feat:increase test coverage
Dataflux fast-listing leverages worksteal algorithm to quickly list objects in a bucket by running several parallel processes.
Adding unit tests that uses storage/emulator to improve test coverage.
Follow up PR to : #10966
Fixes #10731