DMLib Skeleton start #22336
Merged
DMLib Skeleton start #22336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amnguye
requested review from
amishra-dev,
chidozieononiwu,
danieljurek,
kasobol-msft,
mitchdenny,
seanmcc-msft,
tg-msft and
weshaggard
as code owners
kasobol-msft
reviewed
Jul 2, 2021
| public class BlobDirectoryClient | ||
| { | ||
| /// <summary> | ||
| /// The blob's primary <see cref="Uri"/> endpoint. |
There was a problem hiding this comment.
please update the docs for uris.
after looking at next doc string. maybe just make them all "TODO" until api is finalized? so we don't miss updating them
There was a problem hiding this comment.
I actually just copy and pasted from BlobBaseClient, so we kinda have to go and update the docs for all the clients now.
sdk/storage/Azure.Storage.DataMovement/src/Models/CopyMethod.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.DataMovement/src/StorageTransferJob.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.DataMovement/src/StorageTransferManager.cs
Outdated
Show resolved
Hide resolved
Comment on lines
116
to
132
| /// <summary> | ||
| /// Pause transfers. | ||
| /// </summary> | ||
| /// TODO: Returns actual object, or at least in a designated log | ||
| /// file we have a place where people can continue transfers | ||
| public static void PauseTransfer() | ||
| { | ||
| } | ||
|
|
||
| /// <summary> | ||
| /// Cancel Transfers | ||
| /// </summary> | ||
| public static void CancelTransfer() | ||
| { | ||
| // This would remove all transfers from the queue and not log the current progress | ||
| // to the file. Maybe we would also remove the file too as a part of cleanup. | ||
| } |
There was a problem hiding this comment.
The docs say it's plural but API is singular.
Should these be static? I.e. are they going to pause/cancel transfers on all instances of StorageTransferManager?
Maybe we'll eventually need instance level and global level methods like this. But should we start with instance level?
sdk/storage/Azure.Storage.DataMovement/src/TransferItemScheduler.cs
Outdated
Show resolved
Hide resolved
rnpmsft
reviewed
Jul 2, 2021
sdk/storage/Azure.Storage.DataMovement/src/TransferItemScheduler.cs
Outdated
Show resolved
Hide resolved
kasobol-msft
reviewed
Jul 7, 2021
sdk/storage/Azure.Storage.Blobs/api/Azure.Storage.Blobs.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.Blobs/api/Azure.Storage.Blobs.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.Common/api/Azure.Storage.Common.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.DataMovement/api/Azure.Storage.DataMovement.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.DataMovement/api/Azure.Storage.DataMovement.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.DataMovement/api/Azure.Storage.DataMovement.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
Comment on lines
+419
to
+425
| public BlobDirectoryCopyFromUriOptions() { } | ||
| public Azure.Storage.Blobs.Models.AccessTier? AccessTier { get { throw null; } set { } } | ||
| public Azure.Storage.Blobs.Models.BlobDirectoryRequestConditions DestinationConditions { get { throw null; } set { } } | ||
| public System.Collections.Generic.IDictionary<string, string> Metadata { get { throw null; } set { } } | ||
| public System.IProgress<Azure.Response<Azure.Storage.Blobs.Models.BlobCopyInfo>> ProgressHandler { get { throw null; } set { } } | ||
| public Azure.Storage.Blobs.Models.RehydratePriority? RehydratePriority { get { throw null; } set { } } | ||
| public Azure.Storage.Blobs.Models.BlobDirectoryRequestConditions SourceConditions { get { throw null; } set { } } |
There was a problem hiding this comment.
How do these properties work for directory (apart from XConditions)? each blob is going to inherit them?
There was a problem hiding this comment.
When each CopyFromUri call is made for each blob the options will be applied to each call.
There was a problem hiding this comment.
I wonder if we should offer this upfront before there's demand for this.
There was a problem hiding this comment.
or maybe we should group them under DestinationBlobProperties or something. We can wait and see what feedback this gets from larger audience though.
sdk/storage/Azure.Storage.Blobs/api/Azure.Storage.Blobs.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
sdk/storage/Azure.Storage.Common/api/Azure.Storage.Common.netstandard2.0.cs
Outdated
Show resolved
Hide resolved
kasobol-msft
approved these changes
Jul 20, 2021
| @@ -1417,6 +1438,27 @@ public partial class BlobBaseClient | |||
| protected virtual Azure.Storage.Blobs.Specialized.BlobBaseClient WithSnapshotCore(string snapshot) { throw null; } | |||
| public virtual Azure.Storage.Blobs.Specialized.BlobBaseClient WithVersion(string versionId) { throw null; } | |||
| } | |||
| public partial class BlobDirectoryClient | |||
There was a problem hiding this comment.
I wonder if this should go to Specialized namespace. We can keep this as open question.
There was a problem hiding this comment.
I think leaving it in the Specialized namespace makes sense, considering we also put BlockBlobClient, AppendBlobClient, and PageBlobClient in the specialized namespace.
|
CI failures is due to generated code not being up to date in the DataMovement branch due to STG78. I will merge this to our data movement branch and then merge main to the data movement branch |
amnguye
added a commit
that referenced
this pull request
Dec 6, 2021
* WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments
amnguye
added a commit
that referenced
this pull request
Dec 5, 2022
* Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (#21492) * Filesystem scanner refactored to non-static implementation (#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * WIP * Directory Upload and Download basic tests work * Test recordings test * Rerecord tests * WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass * WIP - blobtransfermanager * WIP - Moving configuations for DM Blobs * WIP - blobtransferjobs * Updated storage solution file * WIP - pathScanner tests * WIP - champion scenarios * WIP - champ scenarios * WIP - small changes * WIP' * WIP * WIP * Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (#21492) * Filesystem scanner refactored to non-static implementation (#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * Merge main update * WIP * Builds here without Azure.Storage.DataMovement.Blobs * Builds - DMLib common, DMlib blobs, DMlib samples * Added back in blobs tests * BlobTransferScheduler updated, logger updated, plan file updated * API generates * Rerun some tests, attempting to fix some parallel start problems * Resolve bad merge conflicts * DMLib builds but Blobs.Tests does not build * Conversion from internal job to job details * Run exports api update * Update logger information * Changed threadpool method to use inherit TransferScheduler * Remove previous implementation of getting job details, and combine into one * Removing mistake of committed files * Update to Job Plan header * Updating manager detail of API * Add abstract resumeJob to base storagetransfermanager * Update event arguments to have individual ones for each case, update progress handler to basic handler, update copy method * Removed base DM models, made base event args, made protected ctor StorageTransferManager * Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient * change string to uri for local paths, remove unncessary things from blob job properties * WIP - changing job details out, added more champ scenarios regarding progress tracking * Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client * Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files * WIP * Cannot catch exception properly, tear downs upload call * Addressing Arch board comment * Some fixes from merging from main Remove test dependency on AesGcm for datamovement * WIP * Renamed Experimental to DataMovement * Fixed channel blocklist issue * WIP - changing event handler in uploader to trigger block status * Working commit block handler * WIP * Changes to Download and APIs regarding download * Copy Service Job cleanup * WIP - API changes to StorageResource and Controller * WIP * WIP - Aligning blobs API usage * WIP - Added dependenices to Azure.Storage.DataMovement.Test * WIP - Updated APIs to include checkpointing * WIP - ConsumeableStream -> GetConsumerableStream * WIP - make old API structure internal; todo: remove all old APIs * WIP - Remade API for blobs DM, removed CopyMethod * WIP -Update to StorageTransfer event args name * WIP - Removed factory calls, made dervived storage resource types public * Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource * WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes * WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources * WIP - renamed CommitListTYpe to clearer type * WIP - Update to Copy Options api in blockblob storage, and samples * WIP - Updated APIs * WIP - Updated APIs to include offset streams * WIP - Rename writetooffsetoptions with storageresource prefixed * WIP - copy to and from and update to mmp job plan file * Added over the concurrency tuner * Remove ConfigureAwait from samples * WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource * WIP - fixes to event handler, removable of complete transfer check api * WIP - fix to closing stream when reading from local, setting blocklist order before commiting * WIP - tests * WIP - Remove unnecessary APIs and old code * Removing more unnecessary changes and test recordings for old tests * More removal of old test recordings * Removing BlobFolderClient / BlobVirtualDirectoryClient * Ran Export APIs, moved DataTransferExtensions to DataTransfer * ApiView Comments addressed * Renamed from Blobs.DataMovement to DataMovement.Blobs * Ran ExportApis * Updating assemblyinfo datamovement blobs namespace * Move over Storage Resource tests; Made some API corrections * Remove suppression on editorconfig * Added API for creation of blobs storage resource, max chunk size, more tests, fixes * Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy * PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files * Export APIs * PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files * PR Comments - Move unnecessary return xml removed and removed localfilefactory * PR Comments - Removing leftover folder models from BlobVirtualFolderClient * Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur * Fix to directory, and some test changes to use DataTransfer awaitcompletion * Update to tests and adding discovered length optimization * Ignore some tests for now, to push recording in a separate PR * Update readmes * Ignore more tests * Ignore more local directory tests * Temporarily remove nuget package link; readd when link works when package is released * Update snippets to include length Co-authored-by: Rushi Patel <[email protected]>
sofiar-msft
pushed a commit
to sofiar-msft/azure-sdk-for-net
that referenced
this pull request
Dec 7, 2022
* Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * WIP * Directory Upload and Download basic tests work * Test recordings test * Rerecord tests * WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass * WIP - blobtransfermanager * WIP - Moving configuations for DM Blobs * WIP - blobtransferjobs * Updated storage solution file * WIP - pathScanner tests * WIP - champion scenarios * WIP - champ scenarios * WIP - small changes * WIP' * WIP * WIP * Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * Merge main update * WIP * Builds here without Azure.Storage.DataMovement.Blobs * Builds - DMLib common, DMlib blobs, DMlib samples * Added back in blobs tests * BlobTransferScheduler updated, logger updated, plan file updated * API generates * Rerun some tests, attempting to fix some parallel start problems * Resolve bad merge conflicts * DMLib builds but Blobs.Tests does not build * Conversion from internal job to job details * Run exports api update * Update logger information * Changed threadpool method to use inherit TransferScheduler * Remove previous implementation of getting job details, and combine into one * Removing mistake of committed files * Update to Job Plan header * Updating manager detail of API * Add abstract resumeJob to base storagetransfermanager * Update event arguments to have individual ones for each case, update progress handler to basic handler, update copy method * Removed base DM models, made base event args, made protected ctor StorageTransferManager * Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient * change string to uri for local paths, remove unncessary things from blob job properties * WIP - changing job details out, added more champ scenarios regarding progress tracking * Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client * Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files * WIP * Cannot catch exception properly, tear downs upload call * Addressing Arch board comment * Some fixes from merging from main Remove test dependency on AesGcm for datamovement * WIP * Renamed Experimental to DataMovement * Fixed channel blocklist issue * WIP - changing event handler in uploader to trigger block status * Working commit block handler * WIP * Changes to Download and APIs regarding download * Copy Service Job cleanup * WIP - API changes to StorageResource and Controller * WIP * WIP - Aligning blobs API usage * WIP - Added dependenices to Azure.Storage.DataMovement.Test * WIP - Updated APIs to include checkpointing * WIP - ConsumeableStream -> GetConsumerableStream * WIP - make old API structure internal; todo: remove all old APIs * WIP - Remade API for blobs DM, removed CopyMethod * WIP -Update to StorageTransfer event args name * WIP - Removed factory calls, made dervived storage resource types public * Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource * WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes * WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources * WIP - renamed CommitListTYpe to clearer type * WIP - Update to Copy Options api in blockblob storage, and samples * WIP - Updated APIs * WIP - Updated APIs to include offset streams * WIP - Rename writetooffsetoptions with storageresource prefixed * WIP - copy to and from and update to mmp job plan file * Added over the concurrency tuner * Remove ConfigureAwait from samples * WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource * WIP - fixes to event handler, removable of complete transfer check api * WIP - fix to closing stream when reading from local, setting blocklist order before commiting * WIP - tests * WIP - Remove unnecessary APIs and old code * Removing more unnecessary changes and test recordings for old tests * More removal of old test recordings * Removing BlobFolderClient / BlobVirtualDirectoryClient * Ran Export APIs, moved DataTransferExtensions to DataTransfer * ApiView Comments addressed * Renamed from Blobs.DataMovement to DataMovement.Blobs * Ran ExportApis * Updating assemblyinfo datamovement blobs namespace * Move over Storage Resource tests; Made some API corrections * Remove suppression on editorconfig * Added API for creation of blobs storage resource, max chunk size, more tests, fixes * Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy * PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files * Export APIs * PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files * PR Comments - Move unnecessary return xml removed and removed localfilefactory * PR Comments - Removing leftover folder models from BlobVirtualFolderClient * Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur * Fix to directory, and some test changes to use DataTransfer awaitcompletion * Update to tests and adding discovered length optimization * Ignore some tests for now, to push recording in a separate PR * Update readmes * Ignore more tests * Ignore more local directory tests * Temporarily remove nuget package link; readd when link works when package is released * Update snippets to include length Co-authored-by: Rushi Patel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is so we have an idea of how the moving parts come together.
(I guess somewhere I accidently pushed my BlobDirectoryClient in the branch already, let me figure out how to get that to appear in this PR that way we can look at it)Obviously more has to be done, but feel free to call it out anyways as a friendly reminder.
I also probably touched some configurations I probably shouldn't have. There's some temporary packages being called but once we go to implement our own queue or use an existing queue that I probably didnt know about, we can just remove those.