Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow accessing external environments in Delta Lake query runner #17202

Merged
merged 3 commits into from
Apr 27, 2023
Merged

Allow accessing external environments in Delta Lake query runner #17202

merged 3 commits into from
Apr 27, 2023

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented Apr 24, 2023

Description

The main purpose is quickly accessing product test environments using the new DeltaLakeExternalQueryRunnerMain.

Release notes

(x) This is not user-visible or docs only and no release notes are required.

@ebyhr ebyhr added the no-release-notes This pull request does not require release notes entry label Apr 24, 2023
@cla-bot cla-bot bot added the cla-signed label Apr 24, 2023
@github-actions github-actions bot added the delta-lake Delta Lake connector label Apr 24, 2023
@ebyhr ebyhr requested review from krvikash, findinpath and pajaks April 25, 2023 00:16
@ebyhr ebyhr force-pushed the ebi/delta-external-runner branch 2 times, most recently from 9fcc626 to 22000d0 Compare April 26, 2023 08:36
@ebyhr ebyhr requested a review from findepi April 26, 2023 09:01
ebyhr added 2 commits April 26, 2023 20:37
@ebyhr
Move default file metastore from build method in Delta query runner
6396306
@ebyhr
Avoid hard-coded hive.metastore in DeltaLakeGlueQueryRunnerMain
9d37af9 
This allows accessing product test environment with VM options.
Also, rename to DeltaLakeExternalQueryRunnerMain which is similar
to ElasticsearchExternalQueryRunner.
@ebyhr ebyhr force-pushed the ebi/delta-external-runner branch from 22000d0 to 9d37af9 Compare April 26, 2023 11:37
@findepi
Copy link
Member

findepi commented Apr 26, 2023

This allows accessing product test environment with VM options.

I use DevelopmentServer for that

trino/testing/trino-server-dev/etc/catalog/hive.properties

Lines 10 to 11 in 022a266

# Configuration appropriate for Hive as started by product test environment, e.g.
# trino-product-tests-launcher/bin/run-launcher env up --environment singlenode --without-trino

@ebyhr
Copy link
Member Author

ebyhr commented Apr 26, 2023

I don't want to wait for slow DevelopmentServer startup.

@findepi
Copy link
Member

findepi commented Apr 26, 2023

I don't want to wait for slow DevelopmentServer startup.

agreed. that's why i also have a separate profile, containing only the catalogs i work with

findepi
findepi approved these changes Apr 26, 2023
View reviewed changes
Copy link
Member

@findepi findepi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wouldn't want this (see conversation), but whatever.

@ebyhr ebyhr merged commit 881d5b8 into trinodb:master Apr 27, 2023
@ebyhr ebyhr deleted the ebi/delta-external-runner branch April 27, 2023 00:27
@github-actions github-actions bot added this to the 415 milestone Apr 27, 2023
@colebow colebow mentioned this pull request Apr 27, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@findepi findepi findepi approved these changes

@krvikash krvikash krvikash approved these changes

@findinpath findinpath findinpath approved these changes

@pajaks pajaks Awaiting requested review from pajaks

Assignees
No one assigned
Labels
cla-signed delta-lake Delta Lake connector no-release-notes This pull request does not require release notes entry
Milestone
415
Development

Successfully merging this pull request may close these issues.
4 participants
@ebyhr @findepi @krvikash @findinpath