Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document how to to configure multiple azure storage accounts in single catalog using native-azure #23538

Closed
AlakmarShafin opened this issue Sep 24, 2024 · 10 comments
Assignees
Labels

Comments

@AlakmarShafin
Copy link

There is no way to confugure multiple azure storage account in a single delta lake connector when fs.native-azure.enabled=true as core-sites.xml is not used when this config property is true and if we are going to depreciate legacy azure connector there will be no way to do so.

@hashhar
Copy link
Member

hashhar commented Oct 8, 2024

cc: @anusudarsan / @electrum

@anusudarsan
Copy link
Member

Im looking into this. Will update shortly

@anusudarsan anusudarsan self-assigned this Oct 8, 2024
@anusudarsan
Copy link
Member

@AlakmarShafin I just verified that you can use oauth authentication to achieve this .

  1. You will have to create a service principal in Azure (App registrations). Take a note of client secret to use in Trino config later.
  2. For all storage accounts you want to access from Trino, assign the appropriate policies for your service principal.
Select Access Control (IAM) - > Click Add role assignment. -> Select the appropriate Role (e.g., Storage Blob Data Contributor or Storage Account Contributor) -> In the Assign access to dropdown, choose User, group, or service principal ->  Search for the service principal you created -> Save.

  1. Once the Azure set up is complete, in Trino you can use the following config.
fs.native-azure.enabled=true
azure.auth-type=oauth
azure.oauth.endpoint=https://login.microsoftonline.com/<tenantid>/oauth2/token
azure.oauth.client-id=application-clientid
azure.oauth.tenant-id=tenantid
azure.oauth.secret=client secret (this will be from step 1)

@hashhar
Copy link
Member

hashhar commented Oct 8, 2024

cc: @mosabua this might be useful to have in docs (as a how-to maybe?)

Thanks @anusudarsan.

@mosabua
Copy link
Member

mosabua commented Oct 8, 2024

Definitely could be added to the docs in https://trino.io/docs/current/object-storage/file-system-azure.html somewhere. Lets leave this issue open as doc task for whowever wants to send a PR

@mosabua mosabua added the docs label Oct 8, 2024
@mosabua
Copy link
Member

mosabua commented Oct 8, 2024

Question will also be .. does that only work for Delta Lake .. it should work for all formats.

@hashhar hashhar changed the title How to to configure multiple azure storage accounts in single delta lake catalog using native-azure Document how to to configure multiple azure storage accounts in single catalog using native-azure Oct 8, 2024
@anusudarsan
Copy link
Member

@mosabua all formats. nothing specific to any table format here

@anusudarsan
Copy link
Member

I will send a docs PR

@anusudarsan
Copy link
Member

doc PR merged. Closing. @AlakmarShafin let us know if the above suggestion works.

@AlakmarShafin
Copy link
Author

Thanks @anusudarsan for resolving it quickly. we have tried and tested it, it's working perfectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

4 participants