Unify Delta Lake tests' container handling with Hive and Iceberg #11915

mdesmet · 2022-04-12T19:10:10Z

Description

The MinioClient provides additional features on top of the standard S3 api that may be interesting to keep in our toolbox. So it has been integrated the HiveMinioDataLake container setup:

Uploading of files based on resources (similar to below missing HadoopHive feature)
Event notifications on when files are created or updated similar to AWS lambda (not currently used though)

BaseTestContainer has been extended for executing commands in a container and retrieving the result and loading resources.

Is this change a fix, improvement, new feature, refactoring, or other?

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

Change only impacting tests for Hive, Iceberg and Delta Lake and common test libraries.

How would you describe this change to a non-technical end user or system administrator?

N/A

Related issues, pull requests, and links

Fixes Unify Delta Lake tests' container handling with Hive and Iceberg #11300

Documentation

(x) No documentation is needed.

Release notes

(x) No release notes entries required.

...-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAdlsConnectorSmokeTest.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAdlsStorage.java

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAnalyze.java

findinpath · 2022-06-23T03:07:17Z

There would be a few follow-ups (very likely a different PR to avoid this one exploding):

Extract the HiveMinioDataLake and the other classes to trino-testing-containers.
Adapt trino-faulttolerant-tests to drop the dependency and use the infrastructure previously moved to trino-testing-containers instead

trino/testing/trino-faulttolerant-tests/pom.xml

Lines 148 to 153 in 6b70752

    
           <dependency> 
        
               <groupId>io.trino</groupId> 
        
               <artifactId>trino-delta-lake</artifactId> 
        
               <type>test-jar</type> 
        
               <scope>test</scope> 
        
           </dependency>

findinpath · 2022-06-23T03:09:42Z

Please rebase the PR.

findinpath · 2022-06-23T02:54:44Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/util/PrintingLogConsumer.java

-
-import static java.util.Objects.requireNonNull;
-import static org.testcontainers.containers.output.OutputFrame.OutputType.END;
-


This logging functionality is rather useful.
Let's not drop it.

FYI #12947

findinpath · 2022-06-23T02:55:06Z

plugin/trino-iceberg/pom.xml

@@ -355,6 +355,29 @@
            <scope>test</scope>
        </dependency>

+        <dependency>


minio has been bumped to 7.x in the meantime.

👍 for that. But I assume it's a separate PR to bump this version.

findinpath · 2022-06-23T03:14:28Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

@@ -89,6 +93,7 @@ public void start()
                .build();
        s3Client.createBucket(this.bucketName);


Why do we have both s3client and minioclient?
I think that the s3client is now superfluos.

👍 for that. One or another. My initial idea was to use S3Client as this would allow to have some base class in future which would handle real S3 tests and Minio based test.
For now it seems we should stick to one selected client.

findinpath · 2022-06-23T03:20:04Z

...rino-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeConnectorSmokeTest.java

@@ -118,11 +117,26 @@ abstract QueryRunner createDeltaLakeQueryRunner(Map<String, String> connectorPro

    abstract List<String> listCheckpointFiles(String transactionLogDirectory);

+    protected void copyResources(String resourcePath, String target)
+    {
+        hiveMinioDataLake.getMinioClient().copyResourcePath(bucketName, resourcePath, target);


hiveMinioDataLake.getMinioClient().copyResourcePath keeps appearing multiple times in the PR.

Would it make sense to expose in HiveMinioDataLake (same as it was done in DockerizedMinioDataLake the methods :

copyResources

writeFile

listFiles

?

findinpath · 2022-06-23T03:24:59Z

...o-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeAwsConnectorSmokeTest.java

-                ImmutableMap.of("io/trino/plugin/deltalake/core-site.xml", "/etc/hadoop/conf/core-site.xml"),
-                ImmutableMap.of());
-        return dockerizedMinioDataLake;
+        hiveMinioDataLake = new HiveMinioDataLake(bucketName, ImmutableMap.of(), "ghcr.io/trinodb/testing/hdp2.6-hive");


Pls avoid using magic strings for the docker image - Let's introduce a constant for this purpose.

NIT: I remember there is a class where we store docker image version.
io.trino.plugin.deltalake.util.DockerImages
Maybe let's extract it to test containers and keep those values there

findinpath · 2022-06-23T03:25:22Z

...-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAdlsConnectorSmokeTest.java

+        HiveMinioDataLake hiveMinioDataLake = new HiveMinioDataLake(
+                bucketName,
+                ImmutableMap.of(hadoopCoreSiteXmlTempFile.normalize().toAbsolutePath().toString(), "/etc/hadoop/conf/core-site.xml"),
+                "ghcr.io/trinodb/testing/hdp3.1-hive");


Pls avoid using magic strings for the docker image - Let's introduce a constant for this purpose.

findinpath · 2022-06-23T03:33:05Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/util/TestingHadoop.java

-        container = Failsafe.with(retryPolicy).get(() -> startHadoopContainer(image, ports, resourcesToMount, filesToMount, extraHosts, network));
-        closer.register(container::stop);
-
-        // HACK: even though Hadoop is accessed by proxy, Hadoop still tries to resolve hadoop-master


@losipiuk I just want to make sure that this logic is not needed with the transition towards HiveHadoop where this setting is not made anymore.

aczajkowski

Nice % few nit comments

aczajkowski · 2022-06-24T10:24:25Z

...ng/trino-testing-containers/src/main/java/io/trino/testing/containers/BaseTestContainer.java

+            }
+        }
+        catch (IOException | InterruptedException e) {
+            throw new RuntimeException("Exception while running command: " + printableCommand(commandAndArgs), e);


NIT: Just use String.join

String.join(" ", commandAndArgs)

aczajkowski · 2022-06-24T10:25:43Z

...ng/trino-testing-containers/src/main/java/io/trino/testing/containers/BaseTestContainer.java

+    public void runCommandInContainer(String... commandAndArgs)
+    {
+        try {
+            Container.ExecResult execResult = container.execInContainer(commandAndArgs);


NIT: call public method from this class to not have redundant code.

Container.ExecResult execResult = executeInContainer(commandAndArgs)

aczajkowski · 2022-06-24T10:29:10Z

...ng/trino-testing-containers/src/main/java/io/trino/testing/containers/BaseTestContainer.java

@@ -76,7 +83,8 @@ protected void setupContainer()
        for (int port : this.ports) {
            container.addExposedPort(port);
        }
-        filesToMount.forEach(this::copyFileToContainer);
+        resourcesToMount.forEach(this::copyFileToContainer);


NIT: If we made that change so let's maybe rename copyFileToContainer to copyResourceToContainer

aczajkowski · 2022-06-24T10:35:09Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

+        HiveHadoop.Builder hiveHadoopBuilder = HiveHadoop.builder()
+                .withImage(hiveHadoopImage)
+                .withNetwork(network);
+        if (hiveHadoopFilesToMount.isEmpty()) {


Not sure here but shouldn't we always copy resources "hive_minio_datalake/hive-core-site.xml"
and if there are any files to mount then also files like

HiveHadoop.Builder hiveHadoopBuilder = HiveHadoop.builder() .withImage(hiveHadoopImage) .withNetwork(network) .withResourcesToMount(ImmutableMap.of("hive_minio_datalake/hive-core-site.xml", "/etc/hadoop/conf/core-site.xml")); if (!hiveHadoopFilesToMount.isEmpty()) { hiveHadoopBuilder.withFilesToMount(hiveHadoopFilesToMount); }

We could potentially just blindly allways add withFilesToMount. It it's empty forEach will not iterate and do not mount them

aczajkowski · 2022-06-24T10:37:04Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

@@ -89,6 +93,7 @@ public void start()
                .build();
        s3Client.createBucket(this.bucketName);


👍 for that. One or another. My initial idea was to use S3Client as this would allow to have some base class in future which would handle real S3 tests and Minio based test.
For now it seems we should stick to one selected client.

aczajkowski · 2022-06-24T10:38:16Z

plugin/trino-iceberg/pom.xml

@@ -355,6 +355,29 @@
            <scope>test</scope>
        </dependency>

+        <dependency>


👍 for that. But I assume it's a separate PR to bump this version.

aczajkowski · 2022-06-24T10:40:38Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/MinioClient.java

@@ -45,7 +45,7 @@
 import static java.util.Objects.requireNonNull;
 import static java.util.regex.Matcher.quoteReplacement;

-class MinioClient
+public class MinioClient


NIT: When we use MinioClient now directly in container. I'm wondering if we could maybe move client creation to MinIo container and fetch it from them anywhere it's needed ?

aczajkowski · 2022-06-24T10:42:04Z

plugin/trino-hive/src/test/resources/hive_minio_datalake/hive-core-site.xml

@@ -24,4 +24,15 @@
        <name>fs.s3.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
+
+    <!-- Hive impersonation -->
+    <property>


Why do we need them now. I can't see changes that might require it ?

Ok I found it in next commit :D

aczajkowski · 2022-06-24T10:44:53Z

...o-delta-lake/src/test/java/io/trino/plugin/deltalake/BaseDeltaLakeAwsConnectorSmokeTest.java

-                ImmutableMap.of("io/trino/plugin/deltalake/core-site.xml", "/etc/hadoop/conf/core-site.xml"),
-                ImmutableMap.of());
-        return dockerizedMinioDataLake;
+        hiveMinioDataLake = new HiveMinioDataLake(bucketName, ImmutableMap.of(), "ghcr.io/trinodb/testing/hdp2.6-hive");


NIT: I remember there is a class where we store docker image version.
io.trino.plugin.deltalake.util.DockerImages
Maybe let's extract it to test containers and keep those values there

aczajkowski · 2022-06-24T10:57:44Z

...-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeAdlsConnectorSmokeTest.java

    }

    @AfterClass(alwaysRun = true)
    public void removeTestData()
    {
-        if (adlsDirectory != null && dockerizedDataLake.getTestingHadoop() != null) {
-            dockerizedDataLake.getTestingHadoop().runCommandInContainer("hadoop", "fs", "-rm", "-f", "-r", adlsDirectory);
+        if (adlsDirectory != null && hiveMinioDataLake.getHiveHadoop() != null) {


i think hiveMinioDataLake.getHiveHadoop() != null is not needed. It will never be null when using HibeMinioDataLake

aczajkowski

LGTM

findinpath · 2022-07-01T07:48:13Z

...-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakePerTransactionMetastoreCache.java


    private final Map<String, Long> hiveMetastoreInvocationCounts = new ConcurrentHashMap<>();

    @AfterClass(alwaysRun = true)
    public final void close()
            throws Exception
    {
-        if (dockerizedMinioDataLake != null) {
-            dockerizedMinioDataLake.close();
+        if (hiveMinioDataLake != null) {


pls create a preparatory commit and remove this command because the dockerized data lake is already torn down in tearDown method.

Gentle reminder for handling this comment.

findinpath · 2022-07-01T07:52:31Z

...-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakePerTransactionMetastoreCache.java

-        deltaLakeProperties.put("hive.s3.aws-secret-key", MINIO_SECRET_KEY);
-        deltaLakeProperties.put("hive.s3.endpoint", dockerizedMinioDataLake.getMinioAddress());
+        deltaLakeProperties.put("hive.metastore.uri", "thrift://" + hiveMinioDataLake.getHiveHadoop().getHiveMetastoreEndpoint());
+        deltaLakeProperties.put("hive.s3.aws-access-key", ACCESS_KEY);


Pls create an extra commit (probably after the present one) to rename ACCESS_KEY to MINIO_ACCESS_KEY and SECRET_KEY to MINIO_SECRET_KEY.

findinpath · 2022-07-01T08:12:06Z

...-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedHiveMetastoreWithViews.java

        assertQuery("SHOW TABLES FROM hive." + schema, "VALUES 'hive_table', 'hive_view', 'delta_table'");

-        assertThatThrownBy(() -> query("SHOW CREATE TABLE delta." + schema + ".hive_table"))
+        assertThatThrownBy(() -> query("SHOW CREATE TABLE delta_lake." + schema + ".hive_table"))


optional: Using io.trino.plugin.deltalake.DeltaLakeQueryRunner#DELTA_CATALOG would be also on option.

We could even omit delta_lake but thought this way is a bit more explicit.

I think it would be better to use just delta as product tests (other connector use same catalog name between normal tests and product tests). No need to change in this PR though.

findinpath · 2022-07-01T08:27:01Z

plugin/trino-hive/src/main/java/io/trino/plugin/hive/s3/TrinoS3FileSystem.java

@@ -1802,7 +1802,7 @@ public static TrinoS3FileSystemStats getFileSystemStats()
        return STATS;
    }

-    private static String getMd5AsBase64(byte[] data, int offset, int length)
+    public static String getMd5AsBase64(byte[] data, int offset, int length)


The visibility of this method doesn't need to be changed in case you'd be using minioClient in HiveMinioDataLake.

findinpath · 2022-07-01T08:32:28Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

@@ -44,6 +59,11 @@
    private State state = State.INITIAL;
    private AmazonS3 s3Client;


I would argue that, at the moment, given that this class is called HiveMinioDataLake and deals (as far as I see) only with MinIO, we can safely drop the AmazonS3 s3Client and use the minioClient instead.

+1 for that. I would stick to one or another. Let's not maintain both.

findinpath · 2022-07-01T08:32:35Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

    public AmazonS3 getS3Client()
    {
        checkState(state == State.STARTED, "Can't provide client when MinIO state is: %s", state);
        return s3Client;
    }

-    public void stop()
-            throws Exception
+    public void copyResources(String resourcePath, String target)


Use minioClient to perform the utility operations that you've reimplemented based on AmazonS3 client. The needed logic exists already in MinioClient class.

findinpath · 2022-07-01T08:36:48Z

testing/trino-testing-containers/src/main/java/io/trino/testing/containers/TestContainers.java

+    {
+        return forClasspathResource(resourcePath)
+                // Container fails to mount jar:file:/<host_path>!<resource_path> resources
+                // This assures that JAR resources are being copied out to tmp locations


assures -> ensures

findinpath

LGTM % comments

@ebyhr CPTAL?

findinpath · 2022-07-05T09:40:38Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

+        HiveHadoop.Builder hiveHadoopBuilder = HiveHadoop.builder()
+                .withImage(hiveHadoopImage)
+                .withNetwork(network);
+        Map<String, String> defaultHiveHadoopResourcesToMount = new HashMap<>(Map.of(


Suggested change

Map<String, String> defaultHiveHadoopResourcesToMount = new HashMap<>(Map.of(

Map<String, String> defaultHiveHadoopResourcesToMount = ImmutableMap.<String, String>builder()

.put("/etc/hadoop/conf/core-site.xml", getPathFromClassPathResource("hive_minio_datalake/hive-core-site.xml"))

.putAll(hiveHadoopFilesToMount)

.buildOrThrow();

I had also tried that, however Guava's ImmutableMap doesn't allow duplicate keys to be inserted. Here we sometimes want to override the core-site.xml with a new one. Hence the standard HashMap.

I suppose you could move the default mount mapping to another constructor. e.g.

public HiveMinioDataLake(String bucketName) { this(bucketName, ImmutableMap.of("/etc/hadoop/conf/core-site.xml", getPathFromClassPathResource("hive_minio_datalake/hive-core-site.xml")), HiveHadoop.DEFAULT_IMAGE); } ...

findinpath · 2022-07-05T09:45:05Z

...-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakePerTransactionMetastoreCache.java


    private final Map<String, Long> hiveMetastoreInvocationCounts = new ConcurrentHashMap<>();

    @AfterClass(alwaysRun = true)
    public final void close()
            throws Exception
    {
-        if (dockerizedMinioDataLake != null) {
-            dockerizedMinioDataLake.close();
+        if (hiveMinioDataLake != null) {


Gentle reminder for handling this comment.

findinpath · 2022-07-05T09:49:27Z

@mdesmet

Remaned ACCESS_KEY and SECRET_KEY to MINIO_ACCESS_KEY and MINIO_SECRET_KEY for readability

Use a shorter commit message header and explain the detail in the commit message content.

e.g. Renamed MinIO credentials constants

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/DeltaLakeQueryRunner.java

ebyhr · 2022-07-06T05:41:24Z

...-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedHiveMetastoreWithViews.java

        assertQuery("SHOW TABLES FROM hive." + schema, "VALUES 'hive_table', 'hive_view', 'delta_table'");

-        assertThatThrownBy(() -> query("SHOW CREATE TABLE delta." + schema + ".hive_table"))
+        assertThatThrownBy(() -> query("SHOW CREATE TABLE delta_lake." + schema + ".hive_table"))


I think it would be better to use just delta as product tests (other connector use same catalog name between normal tests and product tests). No need to change in this PR though.

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

ebyhr · 2022-07-06T05:49:55Z

plugin/trino-hive/src/test/java/io/trino/plugin/hive/containers/HiveMinioDataLake.java

+        HiveHadoop.Builder hiveHadoopBuilder = HiveHadoop.builder()
+                .withImage(hiveHadoopImage)
+                .withNetwork(network);
+        Map<String, String> defaultHiveHadoopResourcesToMount = new HashMap<>(Map.of(


I suppose you could move the default mount mapping to another constructor. e.g.

public HiveMinioDataLake(String bucketName) { this(bucketName, ImmutableMap.of("/etc/hadoop/conf/core-site.xml", getPathFromClassPathResource("hive_minio_datalake/hive-core-site.xml")), HiveHadoop.DEFAULT_IMAGE); } ...

plugin/trino-iceberg/pom.xml

mdesmet · 2022-07-10T21:24:38Z

I have extracted the Builder in a separate commit. I also added catalogName in the DeltaLakeQueryRunner.Builder class as a follow-up on the delta vs delta_lake name discussion. That way the signature of the DeltaLakeQueryRunner factory methods doesn't change too much.

findinpath · 2022-07-11T03:43:59Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/DeltaLakeQueryRunner.java

+                        .put("hive.s3.streaming.part-size", "5MB") //must be at least 5MB according to annotations on io.trino.plugin.hive.s3.HiveS3Config.getS3StreamingPartSize
+                        .putAll(connectorProperties)
+                        .buildOrThrow())
+                .build();


nit: Variable assignment is not needed.

findinpath · 2022-07-11T04:06:03Z

plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/DeltaLakeQueryRunner.java

+
+                queryRunner.installPlugin(new TestingDeltaLakePlugin());
+                Map<String, String> deltaProperties = new HashMap<>(this.deltaProperties.buildOrThrow());
+                if (!deltaProperties.containsKey("hive.metastore.uri")) {


Having business logic in setting up default values for properties is not a good idea.
In case we'll want to handle Glue here, we'll further complicate the logic.
I'd advocate that the client classes of DeltaQueryRunner should specify these properties themselves.

See for reference:

trino/plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedFileMetastoreWithTableRedirections.java

Lines 49 to 50 in 74b324c

.put("hive.metastore", "file")

.put("hive.metastore.catalog.dir", dataDirectory.toString())

trino/plugin/trino-delta-lake/src/test/java/io/trino/plugin/deltalake/TestDeltaLakeSharedGlueMetastoreWithTableRedirections.java

Lines 75 to 76 in 74b324c

.put("hive.metastore", "glue")

.put("hive.metastore.glue.default-warehouse-dir", dataDirectory.toString())

I tend to agree with you. Seems like the IcebergQueryRunner also does this though. I would still consider putting it in the QueryRunner class itself as duplicating this logic everywhere seems a bit redundant. WDYT of adding a builder method withFileMetastore? Seems more explicit and hides the complexity for the clients of the QueryRunner classes.

I would say, let's keep this out of this PR for now.

I'm fine with withFileMetastore because it abstracts the logic of setting the file hive metastore.

let's keep this out of this PR for now.

I would argue that this change is related to the current PR.

Follow-up PR is acceptable to me. Going to merger as-is.

...lake/src/test/java/io/trino/plugin/deltalake/AbstractTestDeltaLakeCreateTableStatistics.java

findinpath · 2022-07-11T09:37:58Z

I find it really great that we have a unified way of dealing with minio data lakes on tests. There's only a bit more left to land this awesome test infrastructure change.

Great job!!

Renamed `ACCESS_KEY` and `SECRET_KEY` to `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` for readability

ebyhr · 2022-07-13T07:33:49Z

Merged, thanks!

cla-bot bot added the cla-signed label Apr 12, 2022

github-actions bot added the tests:hive label Apr 12, 2022

ebyhr reviewed Apr 13, 2022

View reviewed changes

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch 5 times, most recently from 1975b27 to 9d7256b Compare April 14, 2022 21:43

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 9d7256b to e76e463 Compare April 21, 2022 19:47

mdesmet requested a review from findinpath June 22, 2022 17:32

findinpath requested a review from aczajkowski June 23, 2022 03:10

findinpath reviewed Jun 23, 2022

View reviewed changes

aczajkowski reviewed Jun 24, 2022

View reviewed changes

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch 3 times, most recently from 3f4549f to 2309e5f Compare June 27, 2022 11:10

mdesmet requested review from aczajkowski and findinpath June 28, 2022 19:59

aczajkowski approved these changes Jun 29, 2022

View reviewed changes

mdesmet requested a review from ebyhr June 30, 2022 17:16

findinpath reviewed Jul 1, 2022

View reviewed changes

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch 2 times, most recently from 012af24 to 04b6af0 Compare July 4, 2022 19:56

ebyhr mentioned this pull request Jul 4, 2022

Test: Unify Delta Lake tests' container handling with Hive and Iceberg #13080

Closed

Remove duplicated method to tear down test containers

d25d02f

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 04b6af0 to 1c9ba74 Compare July 5, 2022 09:45

findinpath reviewed Jul 5, 2022

View reviewed changes

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 1c9ba74 to 040d015 Compare July 5, 2022 09:46

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 040d015 to 5b2ba76 Compare July 5, 2022 10:10

ebyhr approved these changes Jul 6, 2022

View reviewed changes

Introduce DeltaLakeQueryRunner.Builder

899e648

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 5b2ba76 to c34049d Compare July 10, 2022 21:14

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from c34049d to 86728fc Compare July 11, 2022 00:48

findinpath reviewed Jul 11, 2022

View reviewed changes

mdesmet added 3 commits July 11, 2022 11:45

Unify Delta Lake tests' container handling with Hive and Iceberg

59901af

Renamed MinIO credentials constants

29698f5

Renamed `ACCESS_KEY` and `SECRET_KEY` to `MINIO_ACCESS_KEY` and `MINIO_SECRET_KEY` for readability

Inline DeltaLakeQueryRunner.createS3DeltaLakeQueryRunner

a9c9dd6

mdesmet force-pushed the feature/use_hive_connector_containers_in_tests branch from 86728fc to a9c9dd6 Compare July 11, 2022 09:46

ebyhr merged commit cb703eb into trinodb:master Jul 13, 2022

github-actions bot added this to the 390 milestone Jul 13, 2022

colebow mentioned this pull request Jul 13, 2022

Add Trino 390 release notes #13130

Merged

nemanja-m mentioned this pull request Dec 19, 2022

Add Ceph support for Delta Lake #12728

Closed


		import static java.util.Objects.requireNonNull;
		import static org.testcontainers.containers.output.OutputFrame.OutputType.END;

		@@ -89,6 +93,7 @@ public void start()
		.build();
		s3Client.createBucket(this.bucketName);

		@@ -44,6 +59,11 @@
		private State state = State.INITIAL;
		private AmazonS3 s3Client;

-        Map<String, String> defaultHiveHadoopResourcesToMount = new HashMap<>(Map.of(
+                Map<String, String> defaultHiveHadoopResourcesToMount = ImmutableMap.<String, String>builder()
+                .put("/etc/hadoop/conf/core-site.xml", getPathFromClassPathResource("hive_minio_datalake/hive-core-site.xml"))
+                .putAll(hiveHadoopFilesToMount)
+                .buildOrThrow();

	.put("hive.metastore", "file")
	.put("hive.metastore.catalog.dir", dataDirectory.toString())

	.put("hive.metastore", "glue")
	.put("hive.metastore.glue.default-warehouse-dir", dataDirectory.toString())

Unify Delta Lake tests' container handling with Hive and Iceberg #11915

Unify Delta Lake tests' container handling with Hive and Iceberg #11915

Conversation

mdesmet commented Apr 12, 2022 • edited by ebyhr Loading

Description

Related issues, pull requests, and links

Documentation

Release notes

findinpath commented Jun 23, 2022

findinpath commented Jun 23, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aczajkowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aczajkowski left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Jul 5, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdesmet commented Jul 10, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath Jul 11, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

findinpath commented Jul 11, 2022

ebyhr commented Jul 13, 2022

mdesmet commented Apr 12, 2022 •

edited by ebyhr

Loading

findinpath Jul 11, 2022 •

edited

Loading