DataMovement Initial Review #32483
Merged
DataMovement Initial Review #32483
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…, BreakingChanges stubs
* Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal
…ure/azure-sdk-for-net into feature/storage/data-movement
…ng (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes
* WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments
…ure/azure-sdk-for-net into feature/storage/data-movement
|
/check-enforcer override |
|
/azp run net - storage - tests |
|
/azp run net - storage - ci |
|
Pipelines were unable to run due to time out waiting for the pull request to finish merging. |
1 similar comment
|
Pipelines were unable to run due to time out waiting for the pull request to finish merging. |
|
/check-enforcer override |
|
/azp run net - storage - ci |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/check-enforcer override |
|
/azp run net - storage - ci |
|
Pipelines were unable to run due to time out waiting for the pull request to finish merging. |
|
/check-enforcer override |
|
/azp run net - storage - ci |
|
Pipelines were unable to run due to time out waiting for the pull request to finish merging. |
|
/azp run net - storage - ci |
|
Azure Pipelines successfully started running 1 pipeline(s). |
sofiar-msft
pushed a commit
to sofiar-msft/azure-sdk-for-net
that referenced
this pull request
Dec 7, 2022
* Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * WIP * Directory Upload and Download basic tests work * Test recordings test * Rerecord tests * WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass * WIP - blobtransfermanager * WIP - Moving configuations for DM Blobs * WIP - blobtransferjobs * Updated storage solution file * WIP - pathScanner tests * WIP - champion scenarios * WIP - champ scenarios * WIP - small changes * WIP' * WIP * WIP * Create packages for DM Common and Blobs * Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs * WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus * Initial creation of filesystem scanner for DMLib (Azure#21492) * Filesystem scanner refactored to non-static implementation (Azure#21715) * Created filesystem scanner for DM Common * Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner * Tests remade using native temp files; Scanner now throws errors that precede first successful yield * Changed Posix compatibility dependency * Edited versioning and READMEs to adhere to pipelines * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Refactored scanner to non-static implementation; provided user configurable options for error handling * Removed test dependencies from central package list * Scanner will only work on one path for now * Capitalization on FileSystemScanner * Changed scanner to internal * Refactored FS scanner to use factory model and work better with mocking (Azure#21894) * Refactored FS scanner to use factory model and work better with mocking * Rename class/simplify factory implementation * Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed) * Changed parameter name for scan (continueOnError), re-exported API * More exported API changes * DMLib Skeleton start (Azure#22336) * WIP - Removed DataMovement Blobs package, consildate to one package * WIP - Storage Transfer Jobs * WIP - remove dm blobs * WIP - Added TraansferItemScheduler * Ran exportapis * WIP - Resolve package conflicts * Addressed most PR comments * Ran export-api script * Made job for each specific operation for blobs * Added specific copy directory jobs, added option bags for copy scenarios * Ran ExportApi script * Update comments in StorageTransferManager * Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions * Run ExportAPI * PR Comments * Merge fix * Merge main update * WIP * Builds here without Azure.Storage.DataMovement.Blobs * Builds - DMLib common, DMlib blobs, DMlib samples * Added back in blobs tests * BlobTransferScheduler updated, logger updated, plan file updated * API generates * Rerun some tests, attempting to fix some parallel start problems * Resolve bad merge conflicts * DMLib builds but Blobs.Tests does not build * Conversion from internal job to job details * Run exports api update * Update logger information * Changed threadpool method to use inherit TransferScheduler * Remove previous implementation of getting job details, and combine into one * Removing mistake of committed files * Update to Job Plan header * Updating manager detail of API * Add abstract resumeJob to base storagetransfermanager * Update event arguments to have individual ones for each case, update progress handler to basic handler, update copy method * Removed base DM models, made base event args, made protected ctor StorageTransferManager * Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient * change string to uri for local paths, remove unncessary things from blob job properties * WIP - changing job details out, added more champ scenarios regarding progress tracking * Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client * Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files * WIP * Cannot catch exception properly, tear downs upload call * Addressing Arch board comment * Some fixes from merging from main Remove test dependency on AesGcm for datamovement * WIP * Renamed Experimental to DataMovement * Fixed channel blocklist issue * WIP - changing event handler in uploader to trigger block status * Working commit block handler * WIP * Changes to Download and APIs regarding download * Copy Service Job cleanup * WIP - API changes to StorageResource and Controller * WIP * WIP - Aligning blobs API usage * WIP - Added dependenices to Azure.Storage.DataMovement.Test * WIP - Updated APIs to include checkpointing * WIP - ConsumeableStream -> GetConsumerableStream * WIP - make old API structure internal; todo: remove all old APIs * WIP - Remade API for blobs DM, removed CopyMethod * WIP -Update to StorageTransfer event args name * WIP - Removed factory calls, made dervived storage resource types public * Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource * WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes * WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources * WIP - renamed CommitListTYpe to clearer type * WIP - Update to Copy Options api in blockblob storage, and samples * WIP - Updated APIs * WIP - Updated APIs to include offset streams * WIP - Rename writetooffsetoptions with storageresource prefixed * WIP - copy to and from and update to mmp job plan file * Added over the concurrency tuner * Remove ConfigureAwait from samples * WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource * WIP - fixes to event handler, removable of complete transfer check api * WIP - fix to closing stream when reading from local, setting blocklist order before commiting * WIP - tests * WIP - Remove unnecessary APIs and old code * Removing more unnecessary changes and test recordings for old tests * More removal of old test recordings * Removing BlobFolderClient / BlobVirtualDirectoryClient * Ran Export APIs, moved DataTransferExtensions to DataTransfer * ApiView Comments addressed * Renamed from Blobs.DataMovement to DataMovement.Blobs * Ran ExportApis * Updating assemblyinfo datamovement blobs namespace * Move over Storage Resource tests; Made some API corrections * Remove suppression on editorconfig * Added API for creation of blobs storage resource, max chunk size, more tests, fixes * Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy * PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files * Export APIs * PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files * PR Comments - Move unnecessary return xml removed and removed localfilefactory * PR Comments - Removing leftover folder models from BlobVirtualFolderClient * Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur * Fix to directory, and some test changes to use DataTransfer awaitcompletion * Update to tests and adding discovered length optimization * Ignore some tests for now, to push recording in a separate PR * Update readmes * Ignore more tests * Ignore more local directory tests * Temporarily remove nuget package link; readd when link works when package is released * Update snippets to include length Co-authored-by: Rushi Patel <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TODO:
[x] Remove supressions
[ ] Linking back up Job Plan files to Pause and Resume
[x] Remove remants of BlobFolderClients
[ ] Add back in recordings (in different commit, or else the PR would be even bigger..)