Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python: Alter table plumbing and REST support #6323

Merged
merged 30 commits into from
Jun 22, 2023
Merged

Conversation

Fokko
Copy link
Contributor

@Fokko Fokko commented Nov 30, 2022

The plumbing to do an table.alter().commit() with actions in between. We can reuse this later on to do multi table commits:

from datetime import datetime

now = str(datetime.now())

with Transaction(catalog) as transaction:
    table.alter().setProperties(updated_at=now)
    table.alter().setProperties(updated_at=now)

# Multi table commit done

python/mkdocs/docs/index.md Outdated Show resolved Hide resolved
@Fokko Fokko closed this Feb 28, 2023
@Fokko Fokko deleted the fd-alter-table branch April 26, 2023 09:49
@Fokko Fokko restored the fd-alter-table branch April 26, 2023 09:49
@Fokko Fokko reopened this Apr 26, 2023
@Fokko Fokko force-pushed the fd-alter-table branch 2 times, most recently from d4b6a5b to d7c795c Compare April 28, 2023 10:46
@Fokko Fokko added this to the PyIceberg 0.5.0 release milestone Apr 28, 2023
@Fokko Fokko changed the title Python: Alter table for table properties Python: Alter table plumbing and REST support Apr 28, 2023
python/mkdocs/docs/api.md Outdated Show resolved Hide resolved
python/mkdocs/docs/api.md Outdated Show resolved Hide resolved
python/mkdocs/docs/api.md Outdated Show resolved Hide resolved
@github-actions github-actions bot added the build label Jun 5, 2023
@Fokko Fokko force-pushed the fd-alter-table branch from d285677 to cc6d75f Compare June 6, 2023 21:21
python/mkdocs/docs/api.md Outdated Show resolved Hide resolved
@@ -489,6 +493,39 @@ def rename_table(self, from_identifier: Union[str, Identifier], to_identifier: U

return self.load_table(to_identifier)

def _commit(self, *table_requests: CommitTableRequest) -> CommitTableResponse:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to combine these? Why not have separate commit_table and commit_transaction methods? Then we don't need the check that only one request is supported. I also like that you could previously pass requirements and updates to a public method. Are you trying to restrict access to those in the "public" API for some reason?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, given this is called afterwards in Transaction, it feels more nature to call it just commit_table so we an use self._table.catalog.commit_table instead of self._table.catalog._commit

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, thinking of it, that makes sense. Also, the return type is different for a table and a transaction. Thanks!

@rdblue
Copy link
Contributor

rdblue commented Jun 20, 2023

Looks good to me. We can always update the method signature later since this one is internal. I'd merge, but the lock file has conflicts.

Copy link
Contributor

@jackye1995 jackye1995 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly looks good to me, +1 for merging so we can start adding implementations for the other catalogs.

@@ -489,6 +493,39 @@ def rename_table(self, from_identifier: Union[str, Identifier], to_identifier: U

return self.load_table(to_identifier)

def _commit(self, *table_requests: CommitTableRequest) -> CommitTableResponse:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, given this is called afterwards in Transaction, it feels more nature to call it just commit_table so we an use self._table.catalog.commit_table instead of self._table.catalog._commit


ALWAYS_TRUE = AlwaysTrue()


class Transaction:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we plan to have another MiltiTableTransaction for transactions across tables?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather expect a List[Transaction]

@Fokko
Copy link
Contributor Author

Fokko commented Jun 21, 2023

Thank @jackye1995 for chiming in here! 🙏🏻

@Fokko Fokko merged commit 80aee85 into apache:master Jun 22, 2023
@Fokko Fokko deleted the fd-alter-table branch June 22, 2023 06:56
@Fokko Fokko mentioned this pull request Jun 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants