Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filesystem scanner refactored to non-static implementation #21715

Conversation

rnpmsft
Copy link
Contributor

@rnpmsft rnpmsft commented Jun 8, 2021

All SDK Contribution checklist:

This checklist is used to make sure that common guidelines for a pull request are followed.

  • Please open PR in Draft mode if it is:
    • Work in progress or not intended to be merged.
    • Encountering multiple pipeline failures and working on fixes.
  • If an SDK is being regenerated based on a new swagger spec, a link to the pull request containing these swagger spec changes has been included above.
  • I have read the contribution guidelines.
  • The pull request does not introduce breaking changes.

General Guidelines and Best Practices

  • Title of the pull request is clear and informative.
  • There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

  • Pull request includes test coverage for the included changes.

SDK Generation Guidelines

  • The generate.cmd file for the SDK has been updated with the version of AutoRest, as well as the commitid of your swagger spec or link to the swagger spec, used to generate the code. (Track 2 only)
  • The *.csproj and AssemblyInfo.cs files have been updated with the new version of the SDK. Please double check nuget.org current release version.

Additional management plane SDK specific contribution checklist:

Note: Only applies to Microsoft.Azure.Management.[RP] or Azure.ResourceManager.[RP]

  • Include updated management metadata.
  • Update AzureRP.props to add/remove version info to maintain up to date API versions.

Management plane SDK Troubleshooting

  • If this is very first SDK for a services and you are adding new service folders directly under /SDK, please add new service label and/or contact assigned reviewer.

  • If the check fails at the Verify Code Generation step, please ensure:

    • Do not modify any code in generated folders.
    • Do not selectively include/remove generated files in the PR.
    • Do use generate.ps1/cmd to generate this PR instead of calling autorest directly.
      Please pay attention to the @microsoft.csharp version output after running generate.ps1. If it is lower than current released version (2.3.82), please run it again as it should pull down the latest version.

    Note: We have recently updated the PSH module called by generate.ps1 to emit additional data. This would help reduce/eliminate the Code Verification check error. Please run following command:

      `dotnet msbuild eng/mgmt.proj /t:Util /p:UtilityName=InstallPsModules`
    

Old outstanding PR cleanup

Please note:
If PRs (including draft) has been out for more than 60 days and there are no responses from our query or followups, they will be closed to maintain a concise list for our reviewers.

@rnpmsft rnpmsft requested review from amnguye and kasobol-msft June 8, 2021 21:08
@ghost ghost added the Storage Storage Service (Queues, Blobs, Files) label Jun 8, 2021
@check-enforcer
Copy link

check-enforcer bot commented Jun 8, 2021

This pull request is protected by Check Enforcer.

What is Check Enforcer?

Check Enforcer helps ensure all pull requests are covered by at least one check-run (typically an Azure Pipeline). When all check-runs associated with this pull request pass then Check Enforcer itself will pass.

Why am I getting this message?

You are getting this message because Check Enforcer did not detect any check-runs being associated with this pull request within five minutes. This may indicate that your pull request is not covered by any pipelines and so Check Enforcer is correctly blocking the pull request being merged.

What should I do now?

If the check-enforcer check-run is not passing and all other check-runs associated with this PR are passing (excluding license-cla) then you could try telling Check Enforcer to evaluate your pull request again. You can do this by adding a comment to this pull request as follows:
/check-enforcer evaluate
Typically evaulation only takes a few seconds. If you know that your pull request is not covered by a pipeline and this is expected you can override Check Enforcer using the following command:
/check-enforcer override
Note that using the override command triggers alerts so that follow-up investigations can occur (PRs still need to be approved as normal).

What if I am onboarding a new service?

Often, new services do not have validation pipelines associated with them. In order to bootstrap pipelines for a new service, please perform following steps:

For data-plane/track 2 SDKs Issue the following command as a pull request comment:

/azp run prepare-pipelines
This will run a pipeline that analyzes the source tree and creates the pipelines necessary to build and validate your pull request. Once the pipeline has been created you can trigger the pipeline using the following comment:
/azp run net - [service] - ci

For track 1 management-plane SDKs

Please open a separate PR and to your service SDK path in this file. Once that PR has been merged, you can re-run the pipeline to trigger the verification.

/// <param name="path">Filesystem location.</param>
/// <returns>Enumerable list of absolute paths containing all relevant files the user has permission to access.</returns>
public static IEnumerable<string> ScanLocation(string path)
public IEnumerable<string> FullPaths
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be private?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw some similar track 1 classes had paths public, and followed suit; although, there isn't anything useful added here by making them public. I'll change this to private.

Copy link
Contributor

@kasobol-msft kasobol-msft Jun 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, if there's no reason to make it public then we should make it private. We shouldn't add functionality without need. https://en.wikipedia.org/wiki/You_aren%27t_gonna_need_it

/// Multi-path constructor for FilesystemScanner.
/// </summary>
/// <param name="paths"></param>
public FilesystemScanner(IEnumerable<string> paths)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the scenario somebody would provide collection of paths?
I can imagine somebody could create scanner per path and aggregate results themselves.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The work item for the scanner mentioned maybe allowing one scanner to traverse multiple paths, so I added that here.

If users will be making jobs and adding them to a transfer manager, and scanning is handled by those jobs, then maybe we could have this functionality of using multiple paths be handled by a job instead? In which case, said job would create multiple scanners. Or like you said, users could just make several individual jobs.

In either case, the scanner won't need to worry about this scenario (and it'll probably be easier to build from if not), so I'll let it just worry about a single path.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather build separate component to farm out each directory to different scanner. Or maybe we'll build MultiPathScanner or so.
Ability to scan multiple paths should be added when it's needed - this is when it becomes clear how to do this.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amnguye do we know why would we want to pack multi-path scanning into this thing ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I did put that in the work item. I probably didn't think further when I wrote that because taking in multiple paths would assume the scanner would be stateful.

What I want to attempt to do is have the scanner work on the current path given when it is given a path to scan, and if the user decides to add on more jobs (more paths to scan) then we continue to can the other paths from the queue. However I wanted to finish up adding in the job scheduler / transfer manager so that way we have an API to be able to funnel in these paths to use the scanner.

I'm comfortable getting rid of this API but eventually we will probably add an advanced version of this where it continues to scan, if there's paths to scan.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if we're doing multiple upload jobs with one transfer manager, we want all those jobs to use one scanner instead of one scanner per job? Then, are we waiting for all paths to be scanned and then hitting the file scheduler with all files, or taking a path, sending files to scheduler, taking a path, sending files to scheduler, so on? I'm confused on how we're separating the files of each job, and whether that's even a concern.

Would it be better suited to change functionality from storing paths in instance variables and scanning them on call, to instead keeping the found files in an instance variable and adding to it by adding paths?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it will depend on how many threads we dedicate to the scanner portion of the process, which will determine how many scanners we have. So each job wont use the same scanner, unless we have more than one job. We are not waiting for all paths to be scanned before hitting the file scheduler, but we can implement that part in more detail once we get the file scheduler rolling.

Yes, eventually we should change it so that we are pushing these paths to the file scheduler so that we can start the upload and/or download operation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's make Scanner take one path in ctor for now and make it simple. We'll revisit that choice later when we build upper layers.

@@ -5,108 +5,142 @@
using System.Collections.Generic;
using System.IO;
using System.Text;
using System.Linq;

namespace Azure.Storage.Common.DataMovement
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did we come up with this namespace and package name?
Are we going to have DataMovement per storage service + common?
If so then should this class be internal shared source?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DMLib is more meant to be an extension package, that's how I came up with the namespace. But I'm open to any suggestions about this.

Yes the plan is to have it per storage service (exception of Queues and Blob Batch).

I don't think this should be an internal shared source. Maybe most of the classes will be internal but some of it like classes like TransferManager and the progress handler / pause and resume will be public. I also wanted to cover the scenario if a customer wants to add File Share Jobs along with Blob Jobs and/or datalake jobs.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I definitely think we should engage the Arch Board on this and see what they think.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@amnguye in what scenario would we expose FilesystemScanner publicly ?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.. I definitely misread your last question woops (I was thinking the namespace for some wacky reason).

You're right FileSystemScanner should be internal.

Copy link
Contributor

@kasobol-msft kasobol-msft Jun 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great. I guess when we get to the point where we work on publicly available surface the commons shape will finalize. We're good for now. As for class let's make it internal.

@@ -11,44 +11,58 @@ namespace Azure.Storage.Common.DataMovement
/// <summary>
/// FilesystemScanner class.
/// </summary>
public static class FilesystemScanner
public class FilesystemScanner
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: FileSystemScanner

Copy link
Contributor

@kasobol-msft kasobol-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please make scanner internal class.

@amnguye amnguye merged commit b815966 into Azure:feature/storage/data-movement Jun 10, 2021
amnguye pushed a commit that referenced this pull request Dec 6, 2021
* Created filesystem scanner for DM Common

* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner

* Tests remade using native temp files; Scanner now throws errors that precede first successful yield

* Changed Posix compatibility dependency

* Edited versioning and READMEs to adhere to pipelines

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Scanner will only work on one path for now

* Capitalization on FileSystemScanner

* Changed scanner to internal
amnguye added a commit that referenced this pull request Dec 5, 2022
* Create packages for DM Common and Blobs

* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs

* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus

* Initial creation of filesystem scanner for DMLib (#21492)

* Filesystem scanner refactored to non-static implementation  (#21715)

* Created filesystem scanner for DM Common

* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner

* Tests remade using native temp files; Scanner now throws errors that precede first successful yield

* Changed Posix compatibility dependency

* Edited versioning and READMEs to adhere to pipelines

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Scanner will only work on one path for now

* Capitalization on FileSystemScanner

* Changed scanner to internal

* Refactored FS scanner to use factory model and work better with mocking (#21894)

* Refactored FS scanner to use factory model and work better with mocking

* Rename class/simplify factory implementation

* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)

* Changed parameter name for scan (continueOnError), re-exported API

* More exported API changes

* DMLib Skeleton start (#22336)

* WIP - Removed DataMovement Blobs package, consildate to one package

* WIP - Storage Transfer Jobs

* WIP - remove dm blobs

* WIP - Added TraansferItemScheduler

* Ran exportapis

* WIP - Resolve package conflicts

* Addressed most PR comments

* Ran export-api script

* Made job for each specific operation for blobs

* Added specific copy directory jobs, added option bags for copy scenarios

* Ran ExportApi script

* Update comments in StorageTransferManager

* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions

* Run ExportAPI

* PR Comments

* Merge fix

* WIP

* Directory Upload and Download basic tests work

* Test recordings test

* Rerecord tests

* WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass

* WIP - blobtransfermanager

* WIP - Moving configuations for DM Blobs

* WIP - blobtransferjobs

* Updated storage solution file

* WIP - pathScanner tests

* WIP - champion scenarios

* WIP - champ scenarios

* WIP - small changes

* WIP'

* WIP

* WIP

* Create packages for DM Common and Blobs

* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs

* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus

* Initial creation of filesystem scanner for DMLib (#21492)

* Filesystem scanner refactored to non-static implementation  (#21715)

* Created filesystem scanner for DM Common

* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner

* Tests remade using native temp files; Scanner now throws errors that precede first successful yield

* Changed Posix compatibility dependency

* Edited versioning and READMEs to adhere to pipelines

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Scanner will only work on one path for now

* Capitalization on FileSystemScanner

* Changed scanner to internal

* Refactored FS scanner to use factory model and work better with mocking (#21894)

* Refactored FS scanner to use factory model and work better with mocking

* Rename class/simplify factory implementation

* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)

* Changed parameter name for scan (continueOnError), re-exported API

* More exported API changes

* DMLib Skeleton start (#22336)

* WIP - Removed DataMovement Blobs package, consildate to one package

* WIP - Storage Transfer Jobs

* WIP - remove dm blobs

* WIP - Added TraansferItemScheduler

* Ran exportapis

* WIP - Resolve package conflicts

* Addressed most PR comments

* Ran export-api script

* Made job for each specific operation for blobs

* Added specific copy directory jobs, added option bags for copy scenarios

* Ran ExportApi script

* Update comments in StorageTransferManager

* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions

* Run ExportAPI

* PR Comments

* Merge fix

* Merge main update

* WIP

* Builds here without Azure.Storage.DataMovement.Blobs

* Builds - DMLib common, DMlib blobs, DMlib samples

* Added back in blobs tests

* BlobTransferScheduler updated, logger updated, plan file updated

* API generates

* Rerun some tests, attempting to fix some parallel start problems

* Resolve bad merge conflicts

* DMLib builds but Blobs.Tests does not build

* Conversion from internal job to job details

* Run exports api update

* Update logger information

* Changed threadpool method to use inherit TransferScheduler

* Remove previous implementation of getting job details, and combine into one

* Removing mistake of committed files

* Update to Job Plan header

* Updating manager detail of API

* Add abstract resumeJob to base storagetransfermanager

* Update event arguments to have individual ones for each case, update progress handler to basic handler,  update copy method

* Removed base DM models, made base event args, made protected ctor StorageTransferManager

* Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient

* change string to uri for local paths, remove unncessary things from blob job properties

* WIP - changing job details out, added more champ scenarios regarding progress tracking

* Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client

* Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files

* WIP

* Cannot catch exception properly, tear downs upload call

* Addressing Arch board comment

* Some fixes from merging from main

Remove test dependency on AesGcm for datamovement

* WIP

* Renamed Experimental to DataMovement

* Fixed channel blocklist issue

* WIP - changing event handler in uploader to trigger block status

* Working commit block handler

* WIP

* Changes to Download and APIs regarding download

* Copy Service Job cleanup

* WIP - API changes to StorageResource and Controller

* WIP

* WIP - Aligning blobs API usage

* WIP - Added dependenices to Azure.Storage.DataMovement.Test

* WIP - Updated APIs to include checkpointing

* WIP - ConsumeableStream -> GetConsumerableStream

* WIP - make old API structure internal; todo: remove all old APIs

* WIP - Remade API for blobs DM, removed CopyMethod

* WIP -Update to StorageTransfer event args name

* WIP - Removed factory calls, made dervived storage resource types public

* Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource

* WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes

* WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources

* WIP - renamed CommitListTYpe to clearer type

* WIP - Update to Copy Options api in blockblob storage, and samples

* WIP - Updated APIs

* WIP - Updated APIs to include offset streams

* WIP - Rename writetooffsetoptions with storageresource prefixed

* WIP - copy to and from and update to mmp job plan file

* Added over the concurrency tuner

* Remove ConfigureAwait from samples

* WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource

* WIP - fixes to event handler, removable of complete transfer check api

* WIP - fix to closing stream when reading from local, setting blocklist order before commiting

* WIP - tests

* WIP - Remove unnecessary APIs and old code

* Removing more unnecessary changes and test recordings for old tests

* More removal of old test recordings

* Removing BlobFolderClient / BlobVirtualDirectoryClient

* Ran Export APIs, moved DataTransferExtensions to DataTransfer

* ApiView Comments addressed

* Renamed from Blobs.DataMovement to DataMovement.Blobs

* Ran ExportApis

* Updating assemblyinfo datamovement blobs namespace

* Move over Storage Resource tests; Made some API corrections

* Remove suppression on editorconfig

* Added API for creation of blobs storage resource, max chunk size, more tests, fixes

* Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy

* PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files

* Export APIs

* PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files

* PR Comments - Move unnecessary return xml removed and removed localfilefactory

* PR Comments - Removing leftover folder models from BlobVirtualFolderClient

* Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur

* Fix to directory, and some test changes to use DataTransfer awaitcompletion

* Update to tests and adding discovered length optimization

* Ignore some tests for now, to push recording in a separate PR

* Update readmes

* Ignore more tests

* Ignore more local directory tests

* Temporarily remove nuget package link; readd when link works when package is released

* Update snippets to include length

Co-authored-by: Rushi Patel <[email protected]>
sofiar-msft pushed a commit to sofiar-msft/azure-sdk-for-net that referenced this pull request Dec 7, 2022
* Create packages for DM Common and Blobs

* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs

* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus

* Initial creation of filesystem scanner for DMLib (Azure#21492)

* Filesystem scanner refactored to non-static implementation  (Azure#21715)

* Created filesystem scanner for DM Common

* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner

* Tests remade using native temp files; Scanner now throws errors that precede first successful yield

* Changed Posix compatibility dependency

* Edited versioning and READMEs to adhere to pipelines

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Scanner will only work on one path for now

* Capitalization on FileSystemScanner

* Changed scanner to internal

* Refactored FS scanner to use factory model and work better with mocking (Azure#21894)

* Refactored FS scanner to use factory model and work better with mocking

* Rename class/simplify factory implementation

* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)

* Changed parameter name for scan (continueOnError), re-exported API

* More exported API changes

* DMLib Skeleton start (Azure#22336)

* WIP - Removed DataMovement Blobs package, consildate to one package

* WIP - Storage Transfer Jobs

* WIP - remove dm blobs

* WIP - Added TraansferItemScheduler

* Ran exportapis

* WIP - Resolve package conflicts

* Addressed most PR comments

* Ran export-api script

* Made job for each specific operation for blobs

* Added specific copy directory jobs, added option bags for copy scenarios

* Ran ExportApi script

* Update comments in StorageTransferManager

* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions

* Run ExportAPI

* PR Comments

* Merge fix

* WIP

* Directory Upload and Download basic tests work

* Test recordings test

* Rerecord tests

* WIP - not all ListBlobs/GetBlobs tests for DirectoryClient pass

* WIP - blobtransfermanager

* WIP - Moving configuations for DM Blobs

* WIP - blobtransferjobs

* Updated storage solution file

* WIP - pathScanner tests

* WIP - champion scenarios

* WIP - champ scenarios

* WIP - small changes

* WIP'

* WIP

* WIP

* Create packages for DM Common and Blobs

* Making Test packages for DM Common and Blobs; Added Readme, Changelog, BreakingChanges stubs

* WIP - Added BlobDirectoryUploadOptions and StorageTransferStatus

* Initial creation of filesystem scanner for DMLib (Azure#21492)

* Filesystem scanner refactored to non-static implementation  (Azure#21715)

* Created filesystem scanner for DM Common

* Modifed scanner to properly handle missing permissions; added test cases for filesystem scanner

* Tests remade using native temp files; Scanner now throws errors that precede first successful yield

* Changed Posix compatibility dependency

* Edited versioning and READMEs to adhere to pipelines

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Refactored scanner to non-static implementation; provided user configurable options for error handling

* Removed test dependencies from central package list

* Scanner will only work on one path for now

* Capitalization on FileSystemScanner

* Changed scanner to internal

* Refactored FS scanner to use factory model and work better with mocking (Azure#21894)

* Refactored FS scanner to use factory model and work better with mocking

* Rename class/simplify factory implementation

* Return folders as well; preview of ctor logic changes (only throw if path nonexistent/malformed)

* Changed parameter name for scan (continueOnError), re-exported API

* More exported API changes

* DMLib Skeleton start (Azure#22336)

* WIP - Removed DataMovement Blobs package, consildate to one package

* WIP - Storage Transfer Jobs

* WIP - remove dm blobs

* WIP - Added TraansferItemScheduler

* Ran exportapis

* WIP - Resolve package conflicts

* Addressed most PR comments

* Ran export-api script

* Made job for each specific operation for blobs

* Added specific copy directory jobs, added option bags for copy scenarios

* Ran ExportApi script

* Update comments in StorageTransferManager

* Rename BlobUploadDirectoryOptions -> BlobDirectoryUploadOptions

* Run ExportAPI

* PR Comments

* Merge fix

* Merge main update

* WIP

* Builds here without Azure.Storage.DataMovement.Blobs

* Builds - DMLib common, DMlib blobs, DMlib samples

* Added back in blobs tests

* BlobTransferScheduler updated, logger updated, plan file updated

* API generates

* Rerun some tests, attempting to fix some parallel start problems

* Resolve bad merge conflicts

* DMLib builds but Blobs.Tests does not build

* Conversion from internal job to job details

* Run exports api update

* Update logger information

* Changed threadpool method to use inherit TransferScheduler

* Remove previous implementation of getting job details, and combine into one

* Removing mistake of committed files

* Update to Job Plan header

* Updating manager detail of API

* Add abstract resumeJob to base storagetransfermanager

* Update event arguments to have individual ones for each case, update progress handler to basic handler,  update copy method

* Removed base DM models, made base event args, made protected ctor StorageTransferManager

* Changed Directory Download to DownloadTo, added overwrite options, updated internal pipeline transfer for directoryclient

* change string to uri for local paths, remove unncessary things from blob job properties

* WIP - changing job details out, added more champ scenarios regarding progress tracking

* Updating Resume API, correcting event arg names, correctly linked internal deep copy for directory client

* Readded upload directory tests with working json files, changed uploadDirectory API return type, Mild changes to some APIs, renamed part files

* WIP

* Cannot catch exception properly, tear downs upload call

* Addressing Arch board comment

* Some fixes from merging from main

Remove test dependency on AesGcm for datamovement

* WIP

* Renamed Experimental to DataMovement

* Fixed channel blocklist issue

* WIP - changing event handler in uploader to trigger block status

* Working commit block handler

* WIP

* Changes to Download and APIs regarding download

* Copy Service Job cleanup

* WIP - API changes to StorageResource and Controller

* WIP

* WIP - Aligning blobs API usage

* WIP - Added dependenices to Azure.Storage.DataMovement.Test

* WIP - Updated APIs to include checkpointing

* WIP - ConsumeableStream -> GetConsumerableStream

* WIP - make old API structure internal; todo: remove all old APIs

* WIP - Remade API for blobs DM, removed CopyMethod

* WIP -Update to StorageTransfer event args name

* WIP - Removed factory calls, made dervived storage resource types public

* Merged BlobDataController to main controller, renamed DataController to Transfermanager, removed ListType from StorageResource

* WIP - Added Checkpointer API, removed unnecessary -1 enum values, updated job plan header bytes

* WIP - removed options from respective derived storage resource calls, added options bag to blob storage resources

* WIP - renamed CommitListTYpe to clearer type

* WIP - Update to Copy Options api in blockblob storage, and samples

* WIP - Updated APIs

* WIP - Updated APIs to include offset streams

* WIP - Rename writetooffsetoptions with storageresource prefixed

* WIP - copy to and from and update to mmp job plan file

* Added over the concurrency tuner

* Remove ConfigureAwait from samples

* WIP - changes to MMF, service to service copy and adding method to pass the token credential or bearer token to storage resource

* WIP - fixes to event handler, removable of complete transfer check api

* WIP - fix to closing stream when reading from local, setting blocklist order before commiting

* WIP - tests

* WIP - Remove unnecessary APIs and old code

* Removing more unnecessary changes and test recordings for old tests

* More removal of old test recordings

* Removing BlobFolderClient / BlobVirtualDirectoryClient

* Ran Export APIs, moved DataTransferExtensions to DataTransfer

* ApiView Comments addressed

* Renamed from Blobs.DataMovement to DataMovement.Blobs

* Ran ExportApis

* Updating assemblyinfo datamovement blobs namespace

* Move over Storage Resource tests; Made some API corrections

* Remove suppression on editorconfig

* Added API for creation of blobs storage resource, max chunk size, more tests, fixes

* Changed GetStorageResources to return a base class of storage resource; fixed bugs with append / sequential operations; Updated copy status handler for async copy

* PR Comments - reverted necessary config files, moved constants to a separate file, rremvoed globalsupression files

* Export APIs

* PR Comments - removed merge mistakes, updated some xml comments, change some option bags, removed blobstorageresourcefactory, removed more globalsupression files

* PR Comments - Move unnecessary return xml removed and removed localfilefactory

* PR Comments - Removing leftover folder models from BlobVirtualFolderClient

* Updating GetProperties comment XML, removing first value from cpu monitor reading, adding try block to delete file when failed download chunks occur

* Fix to directory, and some test changes to use DataTransfer awaitcompletion

* Update to tests and adding discovered length optimization

* Ignore some tests for now, to push recording in a separate PR

* Update readmes

* Ignore more tests

* Ignore more local directory tests

* Temporarily remove nuget package link; readd when link works when package is released

* Update snippets to include length

Co-authored-by: Rushi Patel <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Storage Storage Service (Queues, Blobs, Files)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants