Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Iceberg query runner with MinIO and file metastore #16861

Merged
merged 1 commit into from
Apr 10, 2023
Merged

Add Iceberg query runner with MinIO and file metastore #16861

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 44 additions & 3 deletions plugin/trino-iceberg/src/test/java/io/trino/plugin/iceberg/IcebergQueryRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,13 @@
import com.google.common.io.Resources;
import io.airlift.http.server.testing.TestingHttpServer;
import io.airlift.log.Logger;
import io.airlift.log.Logging;
import io.trino.plugin.hive.containers.HiveHadoop;
import io.trino.plugin.hive.containers.HiveMinioDataLake;
import io.trino.plugin.iceberg.catalog.jdbc.TestingIcebergJdbcServer;
import io.trino.plugin.tpch.TpchPlugin;
import io.trino.testing.DistributedQueryRunner;
import io.trino.testing.containers.Minio;
import io.trino.tpch.TpchTable;
import org.apache.iceberg.catalog.Catalog;
import org.apache.iceberg.rest.DelegatingRestSessionCatalog;
Expand Down Expand Up @@ -205,9 +207,9 @@ public static void main(String[] args)
}
}

public static final class IcebergMinIoHiveMetastoreQueryRunnerMain
public static final class IcebergMinioHiveMetastoreQueryRunnerMain
{
private IcebergMinIoHiveMetastoreQueryRunnerMain() {}
private IcebergMinioHiveMetastoreQueryRunnerMain() {}

public static void main(String[] args)
throws Exception
Expand Down Expand Up @@ -237,7 +239,46 @@ public static void main(String[] args)
.build();

Thread.sleep(10);
Logger log = Logger.get(IcebergMinIoHiveMetastoreQueryRunnerMain.class);
Logger log = Logger.get(IcebergMinioHiveMetastoreQueryRunnerMain.class);
log.info("======== SERVER STARTED ========");
log.info("\n====\n%s\n====", queryRunner.getCoordinator().getBaseUrl());
}
}

public static final class IcebergMinioQueryRunnerMain
{
private IcebergMinioQueryRunnerMain() {}

public static void main(String[] args)
throws Exception
{
Logging.initialize();

String bucketName = "test-bucket";
Minio minio = Minio.builder().build();
minio.start();
minio.createBucket(bucketName);

DistributedQueryRunner queryRunner = IcebergQueryRunner.builder()
.setCoordinatorProperties(Map.of(
"http-server.http.port", "8080"))
.setIcebergProperties(Map.of(
"iceberg.catalog.type", "TESTING_FILE_METASTORE",
"hive.metastore.catalog.dir", "s3://%s/".formatted(bucketName),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional that hive.metastore.catalog.dir is set to be in the very same bucket as the data for the iceberg tables?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's actually required for managed tables:

trino/plugin/trino-hive/src/main/java/io/trino/plugin/hive/metastore/file/FileHiveMetastore.java

Lines 351 to 355 in c7e8a6d

else if (table.getTableType().equals(MANAGED_TABLE.name())) {
if (!(new Path(table.getStorage().getLocation()).toString().contains(tableMetadataDirectory.toString()))) {
throw new TrinoException(HIVE_METASTORE_ERROR, "Table directory must be " + tableMetadataDirectory);
}
}

Since Iceberg tables are external, we could in theory put them elsewhere, but by default, they live in the catalog location. The file hive metastore is designed to store metadata as dot files alongside the table data. For Iceberg, they end up in different directories due to using unique names for Iceberg.

I'm open to revisiting this if you think there's a need, but since this is for testing, I doubt it's worthwhile.

"hive.s3.aws-access-key", MINIO_ACCESS_KEY,
"hive.s3.aws-secret-key", MINIO_SECRET_KEY,
"hive.s3.endpoint", "http://" + minio.getMinioApiEndpoint(),
"hive.s3.path-style-access", "true",
"hive.s3.streaming.part-size", "5MB"))
.setSchemaInitializer(
SchemaInitializer.builder()
.withSchemaName("tpch")
.withClonedTpchTables(TpchTable.getTables())
.build())
.build();

Thread.sleep(10);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the sleep is redundant (at least ATQF won't sleep after creating a query runner, so it works without a sleep in tests).

will remove it in #16714, where i was adding similar runner

Logger log = Logger.get(IcebergMinioQueryRunnerMain.class);
log.info("======== SERVER STARTED ========");
log.info("\n====\n%s\n====", queryRunner.getCoordinator().getBaseUrl());
}
Expand Down