Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WasbHook needs a copy_object method in Airflow #42497

Closed
2 tasks done
kunaljubce opened this issue Sep 26, 2024 · 5 comments · Fixed by #43037
Closed
2 tasks done

WasbHook needs a copy_object method in Airflow #42497

kunaljubce opened this issue Sep 26, 2024 · 5 comments · Fixed by #43037
Assignees
Labels

Comments

@kunaljubce
Copy link
Contributor

Description

Currently we have a copy_object method for S3Hook. It would be beneficial to have a similar method for WasbHook to be able to copy blob objects seamlessly in Azure.

Use case/motivation

To be able to seamlessly copy blob objects from one Blob container to another within the same Azure storage account. We can also add the native support to copy objects across different storage accounts with an optional parameter.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@kunaljubce kunaljubce added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Sep 26, 2024
Copy link

boring-cyborg bot commented Sep 26, 2024

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@kunaljubce
Copy link
Contributor Author

I am working on this, please assign this to me if the feature looks like a reasonable value add.

@kunaljubce
Copy link
Contributor Author

kunaljubce commented Sep 26, 2024

The cross-storage account functionality has an added layer of complexity since, unlike in AWS, Azure implements a 3 tiered folder structure.

AWS: Account (Auth) -> S3 Bucket -> Prefix -> Keys
Azure: Account (Auth) -> Storage Account (Auth) -> Containers -> Blob Prefix -> Blob objects

In Azure, due to the need to have authentication setup at a storage account level, it becomes imperative to create different instances of WasbHook for different storage accounts, which is not the case in AWS with the S3Hook. My plan is to handle this with an optional parameter, wherein users can mention the destination storage account details/hook for the dest storage account and the file copy would happen accordingly.

Please let me know your thoughts!

@Lee-W Lee-W removed the needs-triage label for new issues that we didn't triage yet label Oct 4, 2024
@Lee-W
Copy link
Member

Lee-W commented Oct 4, 2024

roughly read though it. looks like a good idea. I just assigned it to you. Thanks

@kunaljubce
Copy link
Contributor Author

@Lee-W The PR is up for review now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants