New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[storage][azblob]DownloadFile: Download large files serially #21259

Merged

nakulkar-msft merged 19 commits into main from nakulkar/private/downloadSerial1

Aug 17, 2023

Contributor

nakulkar-msft commented Jul 25, 2023

The purpose of this PR is explained in this or a referenced issue.
The PR does not update generated files.
- These files are managed by the codegen framework at Azure/autorest.go.
Tests are included and/or updated for code changes.
Updates to CHANGELOG.md are included.
MIT license headers are included in each file.

============================================================
The existing DownloadFile method writes blocks of a large blob to the file in random order, which is not performant. This change will make DownloadFile method to write serially instead. The blocks are still downloaded parallelly.
This does not change public interface of the API. Also, this change does not impact performance of small blobs (Or blobs which fit in single chunk).

Below table makes a note of time taken to transfer a 10GiB file with existing and proposed change.

Concurrency    Current	        Proposed
Default = 5     3m22.915s	1m8.111s
8               2m22.286s       47.138s
16	        2m19.675s	47.377s

nakulkar-msft requested review from siminsavani-msft, souravgupta-msft, tasherif-msft, jhendrixMSFT, gapra-msft and vibhansa-msft as code owners

July 25, 2023 11:36

github-actions bot added the Storage label

nakulkar-msft added 6 commits

July 26, 2023 14:32


          Move Buffer manager to shared

ae67312


          Serialize downloading to a file

b9e28cd


          Move DownloadFile to new func

52b4c29


          Add special handling for small files

cf18b56


          Fix build

2146ea6


          Fix build

ca217e6

nakulkar-msft force-pushed the nakulkar/private/downloadSerial1 branch from 66e333c to ca217e6 Compare

July 26, 2023 09:07

nakulkar-msft added 3 commits

July 26, 2023 15:10


          Lint error

b5716a9


          Fix tab spaces

5c2996a


          Fix lint again :(

e131aa9

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/blob/client.go Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/blob/client.go Outdated Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/blob/client.go Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/internal/shared/util.go Outdated Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/blob/constants.go Outdated Show resolved Hide resolved

nakulkar-msft added 2 commits

August 1, 2023 15:11


          Address comments

8a2f301


          Update comment

9252f08

nakulkar-msft requested a review from jhendrixMSFT

August 1, 2023 09:49

vibhansa-msft added the AzBlob label

vibhansa-msft reviewed

View reviewed changes

sdk/storage/azblob/blob/client.go Show resolved Hide resolved

vibhansa-msft reviewed

View reviewed changes

sdk/storage/azblob/blob/client.go Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/blob/constants.go Show resolved Hide resolved

jhendrixMSFT reviewed

View reviewed changes

sdk/storage/azblob/internal/shared/batch_transfer.go Show resolved Hide resolved

nakulkar-msft added 2 commits

August 2, 2023 11:00


          Doc comment for default concurrency

174391f


          Fix formatting

b347741

nakulkar-msft requested review from vibhansa-msft and jhendrixMSFT

August 2, 2023 05:58

vibhansa-msft approved these changes

View reviewed changes

souravgupta-msft approved these changes

View reviewed changes

tasherif-msft reviewed

View reviewed changes

sdk/storage/azblob/internal/shared/mmf_windows.go Show resolved Hide resolved

jhendrixMSFT approved these changes

View reviewed changes

nakulkar-msft and others added 2 commits

August 17, 2023 11:52


          Merge branch 'main' into nakulkar/private/downloadSerial1

b24c0e9


          Fix formatting

7b3fcf2

Contributor Author

nakulkar-msft commented Aug 17, 2023

/azp run go - azblob

azure-pipelines bot commented Aug 17, 2023

Azure Pipelines successfully started running 1 pipeline(s).

nakulkar-msft added 2 commits

August 17, 2023 15:36


          Merge remote-tracking branch 'origin/main' into nakulkar/private/down…

e5c8df8

…loadSerial1


          Fix file read performUploadAndDownloadFileTest() method

ef349c3

Seek to zero in performUploadAndDownloadFileTest before reading.

Contributor Author

nakulkar-msft commented Aug 17, 2023

/azp run go - azblob

azure-pipelines bot commented Aug 17, 2023

Azure Pipelines successfully started running 1 pipeline(s).


          Fix testcase TestBasicDoBatchTransfer

718146e

Contributor Author

nakulkar-msft commented Aug 17, 2023

/azp run go - azblob

azure-pipelines bot commented Aug 17, 2023

Azure Pipelines successfully started running 1 pipeline(s).


          Fix testcase

f765310

nakulkar-msft merged commit 1dc804a into main

nakulkar-msft deleted the nakulkar/private/downloadSerial1 branch

August 17, 2023 11:35

Malsourie mentioned this pull request

The downloaded file won't be truncated to correct size after sdk/storage/azblob:v1.2.0 #21995

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

souravgupta-msft souravgupta-msft approved these changes

tasherif-msft tasherif-msft left review comments

vibhansa-msft vibhansa-msft approved these changes

jhendrixMSFT jhendrixMSFT approved these changes

siminsavani-msft Awaiting requested review from siminsavani-msft

gapra-msft Awaiting requested review from gapra-msft gapra-msft is a code owner

Labels