-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WasbHook needs a copy_object method in Airflow #42497
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
I am working on this, please assign this to me if the feature looks like a reasonable value add. |
The cross-storage account functionality has an added layer of complexity since, unlike in AWS, Azure implements a 3 tiered folder structure. AWS: Account (Auth) -> S3 Bucket -> Prefix -> Keys In Azure, due to the need to have authentication setup at a storage account level, it becomes imperative to create different instances of WasbHook for different storage accounts, which is not the case in AWS with the S3Hook. My plan is to handle this with an optional parameter, wherein users can mention the destination storage account details/hook for the dest storage account and the file copy would happen accordingly. Please let me know your thoughts! |
roughly read though it. looks like a good idea. I just assigned it to you. Thanks |
@Lee-W The PR is up for review now. |
Description
Currently we have a
copy_object
method for S3Hook. It would be beneficial to have a similar method for WasbHook to be able to copy blob objects seamlessly in Azure.Use case/motivation
To be able to seamlessly copy blob objects from one Blob container to another within the same Azure storage account. We can also add the native support to copy objects across different storage accounts with an optional parameter.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: