Parallel download #7429
Parallel download #7429
Conversation
| ).ContinueWith( | ||
| t => | ||
| { | ||
| stream.Flush(); |
There was a problem hiding this comment.
Shouldn't this switch between Flush/FlushAsync depending on the async mode?
There was a problem hiding this comment.
Trying to recall, but if this is the case I recall, it really started to do funky things with the return type -- multiple-nested Task<>, and so forth.
Easy enough to experiment with later.
| @@ -114,8 +119,9 @@ internal static class Append | |||
|
|
|||
| internal static class Block | |||
| { | |||
| public const int DefaultParallelUploadCount = 4; // TODO What should the value really be? Can we get rid of it with a different dispatch algorithm? (probably yes) | |||
| public const int DefaultConcurrentTransfersCount = 4; // TODO What should the value really be? Can we get rid of it with a different dispatch algorithm? (probably yes) | |||
There was a problem hiding this comment.
I asked about this and was told the magic number is 5... for no better reason than everybody else uses the magic number 5. 😄
There was a problem hiding this comment.
@JohnRusk .. this is something you'll want to be looking at.
I chose 4 because once upon a time that was the limit for IE without hacking the registry.
There was a problem hiding this comment.
I was talking to John about this and he said that 4 is actually pretty low. Maybe we should up it to 8. This is to increase the amount of time it takes for uploading and at the same time the number is still low enough to be fair to other connections as well. It doesn't have to be in this PR, we can always make another PR to increase this number.
| @@ -668,6 +669,573 @@ protected virtual BlobBaseClient WithSnapshotImpl(string snapshot) | |||
| } | |||
| #endregion Download | |||
|
|
|||
| #region Parallel Download | |||
There was a problem hiding this comment.
Do we need both Download and Parallel Download? Wouldn't Parallel Download suffice for everything?
There was a problem hiding this comment.
My view was that parallel depended on sequential. There's certainly a limited below which sequential is the way to go. Parallel is primarily a large-file thing, both in terms of how it handles memory and in having an extra call to detect the size and determine whether to use the non-parallel approach.
Another point is that parallel download depends on being handed the destination, rather than being returned as part of BlobDownloadInfo (hence why I'm only returning BlobProperties). They have different signatures because of that.
Update examples (Azure#7429) * Copy compute.json and runCommands.json from 2019-03-01 to 2019-07-01 * changes to add publicIpAddressVersion field (Azure#7173) * Add diskEncryptionSet in swagger compute-2019-07 * resolve semantic conflicts * Fix model conflicts * Resolve readme * Resolve readme * Resolve description conflicts * Improve description * Fix spell error * Add some examples. * fix model error * Update examples
Adds parallel download support from blobs given either destination Stream or FileInfo. Some minor renames of parallel transfer options to reflect a more generic view of things.