The downloaded file won't be truncated to correct size after sdk/storage/azblob:v1.2.0

[Malsourie]

Bug Report

• import path of package in question, e.g. github.com/Azure/azure-sdk-for-go/sdk/storage/azblob
• SDK version e.g. v1.2.0
  • Specify the exact commit if possible: [storage][azblob]DownloadFile: Download large files serially #21259
• output of go version: 1.20

• What happened?
  The downloaded file will not be truncated after method DownloadFile under azure-sdk-for-go/sdk/storage/azblob/blockblob/client.go is called. Therefore the md5 and size of the downloaded file does not match the original blob in container.
• What did you expect or want to happen?
  The file size and md5 should be the same between the downloaded file and the original one in the container.
• How can we reproduce it?
  Call the method DownloadFile and compare the sha/md5 and the size of the file.

[vibhansa-msft]

Can you provide exact test case that you are doing. Why do you expect downloaded file to be truncated? downloaded file shall match exactly the one in container by default.

[vibhansa-msft]

Can you also share the file-size in local and on download.

[Malsourie]

Can you provide exact test case that you are doing.

Test environment: Azure Storage Account with a test container.

1. Upload a dummy file, with size 8241066B to the container via Azure Portal
2. Download the file via Azure Portal to compare_directory
3. rename the downloaded file origin
4. Write a simple go script, which uses sdk/storage/azblob v1.1.0 to download that blob to compare_directory, by calling method blockblob.NewClientWithSharedKeyCredential(...).DownloadFile(...)
5. rename the downloaded file 1_1_0
6. do ls -la <compare_directory>

-rw-r--r--   1 user dialout 8241066 Nov 20 07:54 1_1_0
-rw-r--r--   1 user dialout 8241066 Nov 20 07:50 origin

They have the same size, everything works as expected.
7. Then use the same go script, but update the sdk/storage/azblob to v1.2.0, then call method blockblob.NewClientWithSharedKeyCredential(...).DownloadFile(...) to download the same blob
8. rename the downloaded file 1_2_0
9. do ls -la <compare_directory>

-rw-r--r--   1 user dialout 8241066 Nov 20 07:54 1_1_0
-rw-r--r--   1 user dialout 8388608 Nov 20 08:06 1_2_0
-rw-r--r--   1 user dialout 8241066 Nov 20 07:50 origin

As you can see the file downloaded by v1.2.0 has a different size.

Why do you expect downloaded file to be truncated? downloaded file shall match exactly the one in container by default.
As I have seen in the algorithm:

1. In the comment of method DownloadFile inblockblob/client.go, it is announced that The file would be truncated if the size doesn't match.
2. In DownloadFileOptions struct in blob/models.go, there is one option called BlockSize. My assumption is that the algorithm will first allocate blocks and then download the content into blocks in parallel. Since the file size normally will not match num_of_blocks times size_of_block, in the end the algorithm needs to truncate the file to the original size.
3. There seems to be a truncate algorithm in blob/client.go, but does not work in the new version v1.2.0.

[souravgupta-msft]

@Malsourie, thanks for reporting this issue. The fix has been merged and will be part of the next release.