Improve ADF Tool #21
Improve ADF Tool #21
Conversation
tongwu-sh
commented
Mar 18, 2020
tongwu-sh
commented
- Support parallel processing for ADF Tool
- Support cross region source and destination
- Change to use streaming mode without overhead on disk
Feature/multithreads
Feature/multithreads
Merge from master
Fix data transfer issue
src/Fhir.Anonymizer.AzureDataFactoryPipeline/src/FhirBlobConsumer.cs
Outdated
Show resolved
Hide resolved
| Console.WriteLine($"[{stopWatch.Elapsed.ToString()}][tid:{args.CurrentThreadId}]: {processedCount} Completed. {processedErrorCount} Failed. {consumedCount} consume completed."); | ||
| }; | ||
|
|
||
| await executor.ExecuteAsync(CancellationToken.None, false, progress).ConfigureAwait(false); |
There was a problem hiding this comment.
Can/Should we build resiliency against node crash in the middle of a large blob processing (say picking only unprocessed data on restart)?
There was a problem hiding this comment.
Maybe we can add file level retry at ADF pipeline first? Use foreach activity to run anonymizer tool on single file and retry if it failed?
To achieve resume functionality, looks like we need somewhere for partial status, in batch mode, we can do this with additional storage table or ... If we change to use azure function, looks like we can easily leverage the partial status.
There was a problem hiding this comment.
Should we add a story to track this work?
There was a problem hiding this comment.
Create a story to backlog for tracking. Thanks! https://microsofthealth.visualstudio.com/Health/_workitems/edit/73121/
src/Fhir.Anonymizer.AzureDataFactoryPipeline/src/DataFactoryCustomActivity.cs
Show resolved
Hide resolved
