Document OPTIMIZE for Iceberg #10790

findinpath · 2022-01-25T14:03:33Z

No description provided.

mosabua

Let me have a look again after you updated.

docs/src/main/sphinx/connector/iceberg.rst

mosabua · 2022-01-26T02:49:20Z

docs/src/main/sphinx/connector/iceberg.rst

+ALTER TABLE EXECUTE
+^^^^^^^^^^^^^^^^^^^
+
+The connector supports the following commands for use with


I would simplify .. something like this:

The connector supports collapsing files in a table into fewer larger files with identical data. This operation improves performance and reduces disk usage.

All files with a size below the optional file_size_threshold parameter are collapsed. The default value for the threshold is 100MB:

.. code-block:: sql

ALTER TABLE test_table EXECUTE OPTIMIZE

I did reword the paragraph with your suggestion, but kept the reference towards partitioning. I think it is important to mention that the collapsing takes place per partition in case of partitioned tables.

docs/src/main/sphinx/connector/iceberg.rst

findinpath · 2022-01-26T06:41:22Z

docs/src/main/sphinx/connector/iceberg.rst

+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The connector offers :ref:`ALTER TABLE EXECUTE <alter-table-execute>`
+``OPTIMIZE`` command for collapsing files which correspond to the


@findepi please do provide feedback in case that I'm wrong in the assumption that OPTIMIZE touches only the files which correspond to the current snapshot of the table.

yes it reads off of current snapshot

talking about snapshots here is not necessary. Do you specify that UPDATE or SELECT operate on current snapshot? worse, it can be misunderstood, as if OPTIMIZE was replacing current snapshot, or rewriting the files in situ

docs/src/main/sphinx/connector/hive.rst

findepi · 2022-01-27T11:49:44Z

docs/src/main/sphinx/connector/iceberg.rst

+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The connector offers :ref:`ALTER TABLE EXECUTE <alter-table-execute>`
+``OPTIMIZE`` command for collapsing files which correspond to the


yes it reads off of current snapshot

talking about snapshots here is not necessary. Do you specify that UPDATE or SELECT operate on current snapshot? worse, it can be misunderstood, as if OPTIMIZE was replacing current snapshot, or rewriting the files in situ

docs/src/main/sphinx/connector/iceberg.rst

docs/src/main/sphinx/sql/alter-table.rst

findinpath · 2022-01-27T13:28:15Z

talking about snapshots here is not necessary. Do you specify that UPDATE or SELECT operate on current snapshot?

@findepi think that because this command acts only on the current snapshot, it is worth mentioning this aspect. Not mentioning it could imply for a user not knowing the internals of the Iceberg connector that all the existing snapshots of the table are compacted.

findepi · 2022-01-27T14:52:56Z

talking about snapshots here is not necessary. Do you specify that UPDATE or SELECT operate on current snapshot?

@findepi think that because this command acts only on the current snapshot, it is worth mentioning this aspect. Not mentioning it could imply for a user not knowing the internals of the Iceberg connector that all the existing snapshots of the table are compacted.

User not knowing internals of iceberg won't know & won't care about snapshots.
We should keep snapshot conversation as focused & isolated as possible.

findinpath · 2022-01-31T07:04:07Z

User not knowing internals of iceberg won't know & won't care about snapshots.
We should keep snapshot conversation as focused & isolated as possible.

This is the current state that I have:

The connector offers :ref:ALTER TABLE EXECUTE <alter-table-execute>
optimize command for creating a new snapshot of the table
which contains identical data to the current snapshot with the
content merged into potentially fewer larger files.

@findepi I don't know how can I exclude the snapshot reference out of the description above without degrading the description of the command.

findepi · 2022-01-31T12:12:01Z

@findepi I don't know how can I exclude the snapshot reference out of the description above without degrading the description of the command.

How do we talk about inserts, updates and deletes?

findinpath · 2022-02-03T16:31:44Z

@findepi I took out "snapshot" from the documentation. Please do have another look on the PR.

findepi · 2022-02-07T14:23:52Z

@findinpath please rebase, there is a conflict.

jhlodin · 2022-02-07T22:54:36Z

docs/src/main/sphinx/connector/iceberg.rst

+ALTER TABLE EXECUTE optimize
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The connector offers :ref:`ALTER TABLE EXECUTE <alter-table-execute>`
+``optimize`` command for rewriting the current content of the specified


At risk of overriding your work so far, we have a similar section that documents ALTER TABLE EXECUTE for the Hive connector: https://trino.io/docs/current/connector/hive.html#alter-table-execute

For the sake of consistency, we should make these as similar as possible. Are there any differences in implementation on ALTER TABLE EXECUTE on Hive vs on Iceberg?

The optimize procedure is disabled by default, and can be enabled for a catalog with the <catalog name>.non_transactional_optimize_enabled session property.

This property is not required for the Iceberg connector.

Are there any differences in implementation on ALTER TABLE EXECUTE on Hive vs on Iceberg?

@findepi can you please answer this question? I am not familiar enough with the Hive implementation of OPTIMIZE

Are there any differences in implementation on ALTER TABLE EXECUTE on Hive vs on Iceberg?

They are independent. Hive's is a dangerous, non-transactional operation that can, in case of failure, lead to corrupted table state.

Are there any differences in implementation on ALTER TABLE EXECUTE on Hive vs on Iceberg?

They are independent. Hive's is a dangerous, non-transactional operation that can, in case of failure, lead to corrupted table state.

Would the content in https://trino.io/docs/current/connector/hive.html#alter-table-execute make sense here then, just without the warning note about it being a non-transactional operation?

@jhlodin mostly, yes, except for minor wording #10790 (comment)

@findinpath Can you update the contents of this commit to reuse the Hive documentation? Minus the note as mentioned, and with whatever other feedback there is.

I have aligned both Hive and Iceberg documentation regarding optimize command to follow the same structure.
I've taken the liberty of adding along the way a few more usage examples for the command in the Hive documentation.

Looks great, thanks!

docs/src/main/sphinx/sql/alter-table.rst

jhlodin

LGTM pending comments

jhlodin · 2022-02-08T17:15:38Z

docs/src/main/sphinx/connector/iceberg.rst

+The ``optimize`` command is used for rewriting the active content
+of the specified table so that it is merged into potentially
+fewer larger files.
+In case that the table is partitioned, the data compaction
+acts separately on each partition selected for optimization.
+This operation improves read performance and reduces disk usage.
+
+All files with a size below the optional ``file_size_threshold``
+parameter (default value for the threshold is ``100MB``) are
+merged:


I think this introduction is better than what we had before in Hive, can you apply this there as well?

I slightly did a rewording for Hive because the optimize operation applies only for non-transactional tables.

docs/src/main/sphinx/sql/alter-table.rst

jhlodin · 2022-02-14T15:51:41Z

Hey all, are we ready to merge this PR?

docs/src/main/sphinx/sql/alter-table.rst

docs/src/main/sphinx/connector/hive.rst

cla-bot bot added the cla-signed label Jan 25, 2022

findinpath added the docs label Jan 25, 2022

findinpath requested review from mosabua, jhlodin and findepi January 25, 2022 14:03

findinpath force-pushed the docs/iceberg-optimize branch 3 times, most recently from da9659c to cbbedbc Compare January 25, 2022 14:41

mosabua requested changes Jan 26, 2022

View reviewed changes

findinpath force-pushed the docs/iceberg-optimize branch from cbbedbc to 2f5b181 Compare January 26, 2022 06:39

findinpath commented Jan 26, 2022

View reviewed changes

findepi reviewed Jan 27, 2022

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

findepi reviewed Jan 27, 2022

View reviewed changes

findinpath force-pushed the docs/iceberg-optimize branch 2 times, most recently from 988ee44 to 4d24677 Compare January 27, 2022 13:20

findinpath force-pushed the docs/iceberg-optimize branch from 4d24677 to c5b5734 Compare January 27, 2022 13:33

findinpath force-pushed the docs/iceberg-optimize branch from c5b5734 to 147a867 Compare January 31, 2022 06:57

findinpath force-pushed the docs/iceberg-optimize branch from 147a867 to dfe48ea Compare February 3, 2022 15:29

findinpath requested review from findepi and mosabua February 3, 2022 15:30

jhlodin suggested changes Feb 7, 2022

View reviewed changes

findinpath force-pushed the docs/iceberg-optimize branch from dfe48ea to 23c4665 Compare February 8, 2022 07:40

findepi reviewed Feb 8, 2022

View reviewed changes

docs/src/main/sphinx/sql/alter-table.rst Outdated Show resolved Hide resolved

findinpath force-pushed the docs/iceberg-optimize branch from 23c4665 to cc14ad0 Compare February 8, 2022 17:11

jhlodin approved these changes Feb 8, 2022

View reviewed changes

findinpath force-pushed the docs/iceberg-optimize branch 2 times, most recently from 485714d to 7d61f68 Compare February 8, 2022 17:27

findinpath requested a review from findepi February 14, 2022 18:11

losipiuk reviewed Feb 15, 2022

View reviewed changes

docs/src/main/sphinx/sql/alter-table.rst Outdated Show resolved Hide resolved

losipiuk reviewed Feb 15, 2022

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

losipiuk reviewed Feb 15, 2022

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

losipiuk reviewed Feb 15, 2022

View reviewed changes

docs/src/main/sphinx/connector/hive.rst Outdated Show resolved Hide resolved

findinpath added 2 commits February 15, 2022 12:10

Describe '=>' SQL operator

3f81b30

Remove trailing spaces

198485c

findinpath force-pushed the docs/iceberg-optimize branch from 7d61f68 to 3f90332 Compare February 15, 2022 11:21

findinpath requested a review from losipiuk February 15, 2022 11:21

findinpath added 2 commits February 15, 2022 12:26

Move optimize documentation in an own section

413e170

Document OPTIMIZE for Iceberg

db59b49

findinpath force-pushed the docs/iceberg-optimize branch from 3f90332 to db59b49 Compare February 15, 2022 11:27

losipiuk approved these changes Feb 23, 2022

View reviewed changes

losipiuk merged commit 1d74483 into trinodb:master Feb 23, 2022

github-actions bot added this to the 372 milestone Feb 23, 2022

mosabua mentioned this pull request Feb 23, 2022

Add Trino 372 release notes #11094

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document OPTIMIZE for Iceberg #10790

Document OPTIMIZE for Iceberg #10790

findinpath commented Jan 25, 2022

mosabua left a comment

mosabua Jan 26, 2022

findinpath Jan 26, 2022

findinpath Jan 26, 2022

findepi Jan 27, 2022

findepi Jan 27, 2022

findinpath commented Jan 27, 2022

findepi commented Jan 27, 2022

findinpath commented Jan 31, 2022

findepi commented Jan 31, 2022

findinpath commented Feb 3, 2022

findepi commented Feb 7, 2022

jhlodin Feb 7, 2022

findinpath Feb 8, 2022

findinpath Feb 8, 2022

findepi Feb 8, 2022

jhlodin Feb 8, 2022

findepi Feb 8, 2022

jhlodin Feb 8, 2022

findinpath Feb 8, 2022

jhlodin Feb 8, 2022

jhlodin left a comment

jhlodin Feb 8, 2022

findinpath Feb 8, 2022

jhlodin commented Feb 14, 2022

Document OPTIMIZE for Iceberg #10790

Document OPTIMIZE for Iceberg #10790

Conversation

findinpath commented Jan 25, 2022

mosabua left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Jan 27, 2022

findepi commented Jan 27, 2022

findinpath commented Jan 31, 2022

findepi commented Jan 31, 2022

findinpath commented Feb 3, 2022

findepi commented Feb 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhlodin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jhlodin commented Feb 14, 2022