Create external table location for Hive #17920

vlad-lyutenko · 2023-06-15T13:38:38Z

Description

Create directory structure if it's not exists.
external location will be created if writesToNonManagedTablesEnabled flag is set

The same behaviour is on pure Hive as mentioned in #17920 (comment)

Additional context and related issues

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
(x) Release notes are required, with the following suggested text:

With current change - external location will be created for Hive tables if flag hive.non-managed-table-writes-enabled is set,
otherwise exception will raised as it was before.

# Hive
* Create an empty directory if the external location doesn't exist

krvikash

LGTM. nitpick comments only.

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java

vlad-lyutenko · 2023-06-16T10:10:47Z

LGTM. nitpick comments only.

Big thanks, all comments addressed

ebyhr · 2023-06-16T10:14:22Z

I would recommend reading #1277. I don't think we want to allow creating the directory with external_location table property.

krvikash

It seems the commit message is separated into two lines. Could you please make it in one line?

vlad-lyutenko · 2023-06-16T10:22:46Z

I would recommend reading #1277. I don't think we want to allow creating the directory with external_location table property.

@ebyhr thx, for discussion

Maybe we can introduce this change with some config toggle like:
@Config("hive.allow-create-external-table-path")
?

tooptoop4 · 2023-06-17T07:11:53Z

need a way to disable this for following reasons:

creating a table that points to an existing s3 location that contains sensitive data would circumvent table controls. ie create table myschema.sneakytable WITH (external_location = 's3://sensitive_bucket/data_only_other_team_should_see/')
creating a table that changes/deletes/overwrites data on some existing s3 location

tooptoop4

need config to prevent

hashhar · 2023-06-18T03:08:19Z

creating a table that points to an existing s3 location that contains sensitive data would circumvent table controls. ie create table myschema.sneakytable WITH (external_location = 's3://sensitive_bucket/data_only_other_team_should_see/')

For this there is hive.non-managed-table-creates-enabled and hive.non-managed-table-writes-enabled.

This change does allow a user to create a table pointing to a non-existent location though because the directory would get auto-created.

Also I agree with Yuya on that we shouldn't add this. Hive doesn't allow this for example and also because "Create external table target location, if it's not exists." is what a managed table is - not an external table. External tables are supposed to point to a table which already exists but isn't registered. And for allowing arbitrary location for managed tables see concerns in the issue and the two PRs linked from that issue that Yuya has shared.

vlad-lyutenko · 2023-06-19T10:08:09Z

For me this change is more about, synchronise behaviour with s3:

 private void checkExternalPath(HdfsContext context, Path path)
    {
        try {
            if (!isS3FileSystem(context, hdfsEnvironment, path)) {

As far as I understand on s3 we don't have abstraction as directory, so if for example someone wants to create external table, he could just provide any non existing path and we even don't check it (exists or not exists), because this path will be automatically "created" by s3 (path will be just part of the key).

So I'd like to give the same option for HDFS, if we want external table just give trino a path and we will create it.

vlad-lyutenko · 2023-06-19T10:15:04Z

External tables are supposed to point to a table which already exists but isn't registered

I mean, that I could not find code which actually check that table exists on provided external_path, it's just check that path exists (and only for non s3 file systems).

vlad-lyutenko · 2023-06-19T10:22:44Z

I would recommend reading #1277.

Yes and this is good option to allow create managed table with location provided by the user.
But this will require to disallow create external table on path, where no table exists (not just path not exists).

So as soon, as it will be implemented and merged, it will eliminate current change, but mean time current change just synchronise behaviours between s3 and hdfs.

Praveen2112 · 2023-06-19T11:27:32Z

Hive doesn't allow this for example and also because "Create external table target location, if it's not exists."

Last time when I checked with Hive - if create an external table pointing to non-existing directory, hive would create an empty directory

#1277 is about setting the location of the manged table while this PR is about external table - I'm not sure if it would help us here.

Praveen2112

Can we have additional PT coverage where we check the permission of the directory for external tables ? We do have a similar test for hdfs-impersonation

plugin/trino-hive/src/main/java/io/trino/plugin/hive/HiveMetadata.java

plugin/trino-hive/src/test/java/io/trino/plugin/hive/BaseHiveConnectorTest.java

vlad-lyutenko · 2023-08-03T21:56:18Z

. We should instead separate these into two different properties.

so we should introduce smth like external_location_create_if_not_exists (or some more appropriate name)?

vlad-lyutenko · 2023-08-03T22:11:29Z

All new code needs to use TrinoFileSystem.

didn't find how to create directory using TrinoFileSystem,

or we should add method like createDirecotry in TrinoFileSystem ?

Praveen2112 · 2023-08-04T09:07:49Z

We should NOT create a non-managed location, as by definition it’s not managed by us.

@electrum Please correct me if I am wrong, this definition is derived from hive right, so if hive supports creating an empty directory if it doesn't exist shouldn't (for an external table) we support the same. Today we do support inserting data into a new partition based on writesToNonManagedTablesEnabled flag can't we exhibit a similar behavior for create table as well.

This is why we don’t allow writing to external locations by default.

Even in this PR we do hide them behind a flag which is disabled by default.

Praveen2112 · 2023-08-23T07:28:20Z

Discussed offline with @electrum , now external location will be created if writesToNonManagedTablesEnabled flag is set.

Praveen2112 · 2023-08-24T08:10:32Z

/test-with-secrets sha=39aa31743ed6cd83b0fcd785e22c43f450783ceb

github-actions · 2023-08-24T08:20:39Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/5961189431

Praveen2112 · 2023-08-24T09:05:03Z

/test-with-secrets sha=883ae60c59bafb1f4d7dfb6dafb0ad8db2e1c602

github-actions · 2023-08-24T09:32:39Z

The CI workflow run with tests that require additional secrets finished as failure: https://github.com/trinodb/trino/actions/runs/5961868678

Praveen2112 · 2023-08-24T15:48:55Z

There were a few flaky failures, re-ran them, now the pipeline is green

Praveen2112 · 2023-08-24T15:54:08Z

Thanks for working on this.

findepi · 2024-03-04T14:29:42Z

...roduct-tests-launcher/src/main/java/io/trino/tests/product/launcher/suite/suites/Suite2.java

@@ -36,6 +36,7 @@ public List<SuiteTestRun> getTestRuns(EnvironmentConfig config)
        return ImmutableList.of(
                testOnEnvironment(EnvMultinode.class)
                        .withGroups("configured_features", "hdfs_no_impersonation")
+                        .withExcludedTests("io.trino.tests.product.TestImpersonation.testExternalLocationTableCreationSuccess")


why this exclusion?

to be honest, don't remember any specific reason, need to run and check this test

understood. please recover this info & capture it as a code comment.

understood. please recover this info & capture it as a code comment.

got it I didn't want enable hive.non-managed-table-writes-enabled for whole EnvMultinode environment to break other tests,
which is required for this test

cla-bot bot added the cla-signed label Jun 15, 2023

vlad-lyutenko marked this pull request as draft June 15, 2023 13:38

vlad-lyutenko requested review from kokosing, Praveen2112 and krvikash June 15, 2023 14:50

github-actions bot added hive Hive connector tests:hive labels Jun 15, 2023

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch from a7e603c to a8a7670 Compare June 15, 2023 16:20

vlad-lyutenko changed the title ~~WIP Create external table location~~ Create external table location for Hive Jun 15, 2023

vlad-lyutenko marked this pull request as ready for review June 15, 2023 16:24

This comment was marked as off-topic.

Sign in to view

krvikash approved these changes Jun 15, 2023

View reviewed changes

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch from a8a7670 to 12b7e34 Compare June 16, 2023 10:09

krvikash approved these changes Jun 16, 2023

View reviewed changes

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch from 12b7e34 to ad55aa3 Compare June 16, 2023 10:24

tooptoop4 suggested changes Jun 17, 2023

View reviewed changes

Praveen2112 reviewed Jul 3, 2023

View reviewed changes

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch 4 times, most recently from ded99ad to 1a8c89a Compare July 5, 2023 10:22

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch 5 times, most recently from fffebe7 to 89246ff Compare August 15, 2023 11:16

Praveen2112 requested a review from electrum August 23, 2023 02:59

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch from 89246ff to 5442e70 Compare August 23, 2023 08:29

github-actions bot added the docs label Aug 23, 2023

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch 4 times, most recently from a99dad3 to 39aa317 Compare August 23, 2023 13:13

Praveen2112 approved these changes Aug 23, 2023

View reviewed changes

Create an external table location for Hive if it does not exist

883ae60

vlad-lyutenko force-pushed the vlad-lyutenko/create-external-table-location branch from 39aa317 to 883ae60 Compare August 24, 2023 09:03

Praveen2112 merged commit ba2bb9b into trinodb:master Aug 24, 2023

github-actions bot added this to the 425 milestone Aug 24, 2023

colebow mentioned this pull request Aug 24, 2023

Add Trino 425 release notes #18782

Merged

findepi reviewed Mar 4, 2024

View reviewed changes

vlad-lyutenko mentioned this pull request Mar 5, 2024

Add comment explanation for disabling in Suite2 testExternalLocationTableCreationSuccess test #20936

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create external table location for Hive #17920

Create external table location for Hive #17920

vlad-lyutenko commented Jun 15, 2023 •

edited by Praveen2112

Loading

This comment was marked as off-topic.

krvikash left a comment

vlad-lyutenko commented Jun 16, 2023

ebyhr commented Jun 16, 2023

krvikash left a comment

vlad-lyutenko commented Jun 16, 2023

tooptoop4 commented Jun 17, 2023

tooptoop4 left a comment

hashhar commented Jun 18, 2023

vlad-lyutenko commented Jun 19, 2023

vlad-lyutenko commented Jun 19, 2023 •

edited

Loading

vlad-lyutenko commented Jun 19, 2023 •

edited

Loading

Praveen2112 commented Jun 19, 2023

Praveen2112 left a comment

vlad-lyutenko commented Aug 3, 2023 •

edited

Loading

vlad-lyutenko commented Aug 3, 2023 •

edited

Loading

Praveen2112 commented Aug 4, 2023

Praveen2112 commented Aug 23, 2023

Praveen2112 commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 •

edited

Loading

Praveen2112 commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 •

edited

Loading

Praveen2112 commented Aug 24, 2023

Praveen2112 commented Aug 24, 2023

findepi Mar 4, 2024

vlad-lyutenko Mar 4, 2024

findepi Mar 5, 2024

vlad-lyutenko Mar 5, 2024 •

edited

Loading

vlad-lyutenko Mar 5, 2024

Create external table location for Hive #17920

Create external table location for Hive #17920

Conversation

vlad-lyutenko commented Jun 15, 2023 • edited by Praveen2112 Loading

Description

Additional context and related issues

Release notes

This comment was marked as off-topic.

krvikash left a comment

Choose a reason for hiding this comment

vlad-lyutenko commented Jun 16, 2023

ebyhr commented Jun 16, 2023

krvikash left a comment

Choose a reason for hiding this comment

vlad-lyutenko commented Jun 16, 2023

tooptoop4 commented Jun 17, 2023

tooptoop4 left a comment

Choose a reason for hiding this comment

hashhar commented Jun 18, 2023

vlad-lyutenko commented Jun 19, 2023

vlad-lyutenko commented Jun 19, 2023 • edited Loading

vlad-lyutenko commented Jun 19, 2023 • edited Loading

Praveen2112 commented Jun 19, 2023

Praveen2112 left a comment

Choose a reason for hiding this comment

vlad-lyutenko commented Aug 3, 2023 • edited Loading

vlad-lyutenko commented Aug 3, 2023 • edited Loading

Praveen2112 commented Aug 4, 2023

Praveen2112 commented Aug 23, 2023

Praveen2112 commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 • edited Loading

Praveen2112 commented Aug 24, 2023

github-actions bot commented Aug 24, 2023 • edited Loading

Praveen2112 commented Aug 24, 2023

Praveen2112 commented Aug 24, 2023

findepi Mar 4, 2024

Choose a reason for hiding this comment

vlad-lyutenko Mar 4, 2024

Choose a reason for hiding this comment

findepi Mar 5, 2024

Choose a reason for hiding this comment

vlad-lyutenko Mar 5, 2024 • edited Loading

Choose a reason for hiding this comment

vlad-lyutenko Mar 5, 2024

Choose a reason for hiding this comment

vlad-lyutenko commented Jun 15, 2023 •

edited by Praveen2112

Loading

vlad-lyutenko commented Jun 19, 2023 •

edited

Loading

vlad-lyutenko commented Jun 19, 2023 •

edited

Loading

vlad-lyutenko commented Aug 3, 2023 •

edited

Loading

vlad-lyutenko commented Aug 3, 2023 •

edited

Loading

github-actions bot commented Aug 24, 2023 •

edited

Loading

github-actions bot commented Aug 24, 2023 •

edited

Loading

vlad-lyutenko Mar 5, 2024 •

edited

Loading