Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow glue_table_settings attribute on to_iceberg #2469

Closed
frenchytheasian opened this issue Sep 22, 2023 · 2 comments
Closed

Allow glue_table_settings attribute on to_iceberg #2469

frenchytheasian opened this issue Sep 22, 2023 · 2 comments
Labels
enhancement New feature or request

Comments

@frenchytheasian
Copy link
Contributor

frenchytheasian commented Sep 22, 2023

Is your idea related to a problem? Please describe.
Currently, I'm working with Iceberg tables in Athena. When I want to add comments to the table or columns, I have to make a separate boto3 call to glue.

Describe the solution you'd like
wr.s3.to_parquet contains a glue_table_settings parameter which allows users to add a GlueTableSettings object to configure the glue catalog and update data in one function call. I was wondering if its possible or would make sense to do something similar for the to_iceberg function?

I would willing to open a PR and take a stab at this if it makes sense to add.

@frenchytheasian frenchytheasian added the enhancement New feature or request label Sep 22, 2023
@jaidisido
Copy link
Contributor

Hi, unlike s3.to_parquet which makes a Glue call to create a table in the catalog, the athena.to_iceberg API creates the table via an Athena query. It means that it's limited by the parameters exposed here. We expose the additional_table_parameters in the API but comments are not one of those listed. PRs are always welcome though

@frenchytheasian
Copy link
Contributor Author

@jaidisido Ahh I see. Rather than creating a whole glue_table_settings parameter, I think a column_comments parameter might make more sense. I noticed that you can add column comments in the athena sql. I think this could be done by injecting the values of a column comments parameter into the Athena query. I can work on implementing this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants