feat: Use DH S3Instructions to build Iceberg AWS clients #6113

devinrsmith · 2024-09-23T19:58:31Z

This provides a way for users who are responsible for providing AWS / S3 credentials to specify it in a way where Deephaven can own the S3 client building logic for the Iceberg Catalog in additional to our own data access layer.

Note, this does not deprecate DataInstructionsProviderPlugin, as there may be cases where the user is not responsible for providing these credentials, and it is instead provided via the catalog after catalog authorization. See #6191

extensions/iceberg/s3/src/main/java/io/deephaven/iceberg/util/IcebergToolsS3.java

extensions/s3/src/main/java/io/deephaven/extensions/s3/S3ClientFactory.java

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

extensions/s3/src/main/java/io/deephaven/extensions/s3/S3ClientFactory.java

extensions/iceberg/s3/src/main/java/io/deephaven/iceberg/util/IcebergToolsS3.java

malhotrashivam · 2024-10-10T19:52:33Z

extensions/iceberg/s3/src/test/java/io/deephaven/iceberg/util/IcebergLocalStackTest.java

@@ -15,9 +16,11 @@
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.ENDPOINT;
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.SECRET_ACCESS_KEY;

+@Tag("testcontainers")
+@Deprecated


I don't understand why we had to deprecate these?
They might still give us coverage on a lot of smaller cases that Larry tested.
Is it deprecated in a sense that new tests should not be added?

It's mostly that I think we should not add new tests here, and work to migrating them as mentioned in IcebergToolsTest. @lbooker42 has so far owned this layer, but it should be relatively easy to migrate to db_resource, and then we get the benefit of:

No separate container (s3) needed (+ no need to upload files)

Ne need to have custom catalog IcebergTestCatalog

On disk JDBC catalog + on disk warehouse

I see db_resource as mainly a way to test out how well we can interoperate with Iceberg that has been written via different processes (pyiceberg, spark, etc).

For more thorough testing (once we have our own writing support) we should be able to extend SqliteCatalogBase (or create further specialized tests that look similar to it), which can work with any warehouse - currently the same logic is tested out via local disk, minio, and localstack.

I agree, these new catalog testing code is the more comprehensive way for all future tests. Once we can migrate these tests so we don't lose any coverage, we should remove these as well as the IcebergTestCatalog class.
cc: @lbooker42

extensions/iceberg/src/main/java/io/deephaven/iceberg/util/IcebergTools.java

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

py/server/deephaven/experimental/iceberg.py

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

extensions/iceberg/src/main/java/io/deephaven/iceberg/util/IcebergTools.java

malhotrashivam · 2024-10-10T21:01:36Z

extensions/iceberg/s3/src/test/java/io/deephaven/iceberg/util/IcebergLocalStackTest.java

@@ -15,9 +16,11 @@
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.ENDPOINT;
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.SECRET_ACCESS_KEY;

+@Tag("testcontainers")
+@Deprecated


I agree, these new catalog testing code is the more comprehensive way for all future tests. Once we can migrate these tests so we don't lose any coverage, we should remove these as well as the IcebergTestCatalog class.
cc: @lbooker42

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

malhotrashivam · 2024-10-11T16:17:18Z

extensions/iceberg/s3/src/main/java/io/deephaven/iceberg/util/IcebergToolsS3.java

+        CLEANER.register(adapter.catalog(), cleanup);
+        return adapter;


When I had to do something similar for the S3Request objects, Ryan suggested using CleanupReferenceProcessor. You can check that too, if that has any advantages.

The "register" method on Cleaner is very nice. I tried to add a similarly helpful register method to CleanupReferenceProcessor (essentially, creating a reference behind the scenes that ties an object and a cleanup action), and it almost worked... for some reason though, the caller needs to explicitly retain the returned reference, whereas the same limitation does not apply to Cleaner. It could be I was missing some subtle aspect of the Reference stuff - will have convo w/ Ryan.

I've got a separate PR that adds similar functionality to CleanupReferenceProcessor; #6213

lbooker42 · 2024-10-14T22:25:59Z

extensions/iceberg/s3/src/main/java/io/deephaven/iceberg/util/IcebergToolsS3.java

+    public static IcebergCatalogAdapter createAdapter(
+            @Nullable final String name,
+            @NotNull final Map<String, String> properties,
+            @NotNull final Map<String, String> hadoopConfig,


Is hadoopConfig used in an S3-backed adapter with S3 file IO? Can we drop this as a parameter and create an empty config internal to this function?

It's a good question; org.apache.iceberg.aws.s3.S3FileIO does not use hadoop conf. That said, this method is not imposing S3FileIO. It looks like GlueCatalog is technically written in a way where hadoopConf can be passed along if something besides S3FileIO is used... maybe it's possible to use GlueCatalog and not use S3 as the warehouse... for example, maybe some sort of other AWS NFS storage, I'm not sure. I'm going to add more explicit documentation about this.

lbooker42 · 2024-10-14T22:30:41Z

extensions/iceberg/s3/src/test/java/io/deephaven/iceberg/util/IcebergLocalStackTest.java

@@ -15,9 +16,11 @@
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.ENDPOINT;
 import static org.apache.iceberg.aws.s3.S3FileIOProperties.SECRET_ACCESS_KEY;

+@Tag("testcontainers")
+@Deprecated


extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

extensions/iceberg/s3/src/main/java/io/deephaven/iceberg/util/IcebergToolsS3.java

extensions/s3/src/main/java/io/deephaven/extensions/s3/S3ClientFactory.java

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java

jmao-denver · 2024-10-18T23:19:25Z

py/server/deephaven/experimental/iceberg.py

+    In cases where the caller prefers to use Iceberg's AWS properties, the parity of construction logic will be limited
+    to what Deephaven is able to infer; in advanced cases, it's possible that there will be a difference in construction
+    logic between the Iceberg-managed and Deephaven-managed AWS clients which manifests itself as being able to browse
+    Catalog metadata, but not retrieve Table data.
+


A little unclear without an example. Also it might be nice to point out a possible solution for the problem of not being able to read table data.

The problem is that it's a very opened thing, and it's tough to provide generalizabled example.

py/server/deephaven/experimental/iceberg.py

rcaudy

I'm approving, but since auto-merge is enabled leaving this for a subsequent approval.

py/server/deephaven/experimental/iceberg.py

chipkent

Python LGTM

deephaven-internal · 2024-10-29T22:15:24Z

Labels indicate documentation is required. Issues for documentation have been opened:

Community: deephaven/deephaven-docs-community#349

feat: Use DH S3Instructions to build Iceberg AWS clients

8066110

devinrsmith added DocumentationNeeded ReleaseNotesNeeded Release notes are needed labels Sep 23, 2024

devinrsmith added this to the 0.37.0 milestone Sep 23, 2024

devinrsmith self-assigned this Sep 23, 2024

Refactor some shared code

4dc52bd

devinrsmith mentioned this pull request Sep 25, 2024

feat: Added support to specify AWS profile through S3 instructions #6130

Merged

devinrsmith marked this pull request as ready for review October 8, 2024 18:07

devinrsmith requested review from lbooker42, malhotrashivam and rcaudy October 8, 2024 18:07

Merge remote-tracking branch 'upstream/main' into dh-iceberg-s3-client

9c13e93

malhotrashivam reviewed Oct 9, 2024

View reviewed changes

Review response

de78d8a

devinrsmith requested a review from malhotrashivam October 10, 2024 00:45

small changes

28f7643

malhotrashivam reviewed Oct 10, 2024

View reviewed changes

Plumb through to iceberg.py

c8ecd2c

devinrsmith requested review from chipkent and jmao-denver as code owners October 10, 2024 20:18

remove import

3804ef9

malhotrashivam reviewed Oct 10, 2024

View reviewed changes

devinrsmith added 2 commits October 10, 2024 14:15

Refactor, don't have IcebergTools extendable

385ac25

Delay cleanup until catalog is unreachable

b909ff3

malhotrashivam reviewed Oct 11, 2024

View reviewed changes

lbooker42 reviewed Oct 14, 2024

View reviewed changes

Review response

a4d2b5a

malhotrashivam previously approved these changes Oct 15, 2024

View reviewed changes

rcaudy reviewed Oct 18, 2024

View reviewed changes

review response

7da2369

devinrsmith added 2 commits October 18, 2024 11:16

Merge remote-tracking branch 'upstream/main' into dh-iceberg-s3-client

a6f51a4

Update iceberg python documentation

14aa4fb

devinrsmith dismissed malhotrashivam’s stale review via 14aa4fb October 18, 2024 18:37

devinrsmith requested a review from rcaudy October 18, 2024 18:37

rcaudy reviewed Oct 18, 2024

View reviewed changes

extensions/iceberg/s3/src/main/java/io/deephaven/extensions/s3/DeephavenAwsClientFactory.java Outdated Show resolved Hide resolved

devinrsmith enabled auto-merge (squash) October 18, 2024 19:57

jmao-denver reviewed Oct 18, 2024

View reviewed changes

py/server/deephaven/experimental/iceberg.py Outdated Show resolved Hide resolved

devinrsmith added 2 commits October 21, 2024 10:49

Review response

b62debc

Merge remote-tracking branch 'upstream/main' into dh-iceberg-s3-client

11ba983

devinrsmith requested review from jmao-denver and rcaudy October 21, 2024 19:06

chipkent reviewed Oct 23, 2024

View reviewed changes

py/server/deephaven/experimental/iceberg.py Outdated Show resolved Hide resolved

py/server/deephaven/experimental/iceberg.py Outdated Show resolved Hide resolved

devinrsmith added 5 commits October 28, 2024 15:01

Merge remote-tracking branch 'upstream/main' into dh-iceberg-s3-client

6d0bbad

Fix

1340833

changes

790f086

Migrate cleanup to CleanupReferenceProcessor

57b98a4

fix deps for javadoc

e3a9c65

rcaudy reviewed Oct 29, 2024

View reviewed changes

devinrsmith added 2 commits October 29, 2024 12:24

Merge remote-tracking branch 'upstream/main' into dh-iceberg-s3-client

3e78ce8

fixup after merge

bb7e239

devinrsmith requested a review from chipkent October 29, 2024 19:29

chipkent reviewed Oct 29, 2024

View reviewed changes

py/server/deephaven/experimental/iceberg.py Outdated Show resolved Hide resolved

review response

098388f

devinrsmith requested a review from chipkent October 29, 2024 21:44

chipkent approved these changes Oct 29, 2024

View reviewed changes

devinrsmith merged commit 59f226d into deephaven:main Oct 29, 2024
16 checks passed

devinrsmith deleted the dh-iceberg-s3-client branch October 29, 2024 22:15

github-actions bot locked and limited conversation to collaborators Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Use DH S3Instructions to build Iceberg AWS clients #6113

feat: Use DH S3Instructions to build Iceberg AWS clients #6113

devinrsmith commented Sep 23, 2024 •

edited

Loading

malhotrashivam Oct 10, 2024

devinrsmith Oct 10, 2024

malhotrashivam Oct 10, 2024

lbooker42 Oct 14, 2024

malhotrashivam Oct 10, 2024

malhotrashivam Oct 11, 2024

devinrsmith Oct 15, 2024

malhotrashivam Oct 15, 2024

devinrsmith Oct 18, 2024

lbooker42 Oct 14, 2024

devinrsmith Oct 15, 2024

lbooker42 Oct 14, 2024

jmao-denver Oct 18, 2024

devinrsmith Oct 21, 2024

rcaudy left a comment

chipkent left a comment

deephaven-internal commented Oct 29, 2024

		CLEANER.register(adapter.catalog(), cleanup);
		return adapter;

feat: Use DH S3Instructions to build Iceberg AWS clients #6113

feat: Use DH S3Instructions to build Iceberg AWS clients #6113

Conversation

devinrsmith commented Sep 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rcaudy left a comment

Choose a reason for hiding this comment

chipkent left a comment

Choose a reason for hiding this comment

deephaven-internal commented Oct 29, 2024

devinrsmith commented Sep 23, 2024 •

edited

Loading