Include Column Lineage Info for All Queries #23322

behrooz-stripe · 2024-09-06T22:15:05Z

Description

Column lineage info is extremely helpful but is only available for Create View, Table etc. not Select queries. This PR simply expands the support for lineage for other queries as well.

Additional context and related issues

adds column lineage to updateTarget so it'll be available in EventListener without any other plumbing/changes required. Please see assertLineage for testing, it'll force all the select queries to go through the same tests and assumptions.

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

cla-bot · 2024-09-06T22:15:08Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-06T22:16:53Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-06T22:25:41Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T00:02:54Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T00:09:55Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T00:36:17Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T00:37:17Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T19:25:15Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T22:26:39Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T22:29:10Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T22:44:45Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T22:58:26Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-10T22:59:08Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-12T21:14:00Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-12T21:25:27Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-13T19:53:18Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-13T19:58:58Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-13T20:15:37Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-13T22:20:30Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-13T22:30:06Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

behrooz-stripe · 2024-09-16T19:27:23Z

Hi, I work for Stripe and I've double checked on our end and we already should have a Corporate CLA signed. Please verify on your end. Thanks!

mosabua · 2024-09-16T19:50:34Z

@behrooz-stripe you still need to sign the individual contributor license agreement

behrooz-stripe · 2024-09-16T19:56:27Z

@behrooz-stripe you still need to sign the individual contributor license agreement

thanks @mosabua ! I just emailed the individual one too.

wendigo · 2024-09-17T06:22:38Z

@behrooz-stripe Can you squash commits so that the commit is a logical unit of a change that fully compiles on its own?

cla-bot · 2024-09-17T17:01:29Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

cla-bot · 2024-09-17T17:02:15Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

behrooz-stripe · 2024-09-17T17:04:17Z

@wendigo done.

wendigo · 2024-09-17T17:13:18Z

@behrooz-stripe Please drop merge commit and rebase on master and I'll review it

cla-bot · 2024-09-17T17:22:29Z

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

behrooz-stripe · 2024-09-17T17:24:03Z

@behrooz-stripe Please drop merge commit and rebase on master and I'll review it

alright, should be good now! I think I accidentally clicked on sync button in GitHub UI an it committed the merge one.

behrooz-stripe · 2024-09-17T19:52:49Z

@wendigo seems like after I rebased this testCacheFileOperations is failing but I can't seem to be able to reproduce it locally and it succeeds. Could it be it's flakey?

behrooz-stripe · 2024-09-17T20:28:53Z

nvm, I can see the master is failing with the same error too.

behrooz-stripe · 2024-09-23T18:32:46Z

Hey @wendigo would you please take a look at this PR and provide feedback? thanks!

wendigo · 2024-10-07T16:28:40Z

@behrooz-stripe sorry I don't have context here

behrooz-stripe · 2024-10-07T16:35:08Z

@electrum might have the best context on this based on discussions in this thread: https://trinodb.slack.com/archives/CP1MUNEUX/p1715720365582019

patstevens4 · 2024-10-28T09:43:52Z

Bump, eagerly waiting for this feature! Thanks for all the work on this great idea

hashhar · 2024-11-18T13:19:55Z

@Praveen2112 PTAL. This looks good to go in my opinion but you're the expert here.

behrooz-stripe · 2024-12-03T18:40:53Z

Bump @Praveen2112 @hashhar

github-actions · 2024-12-25T17:03:16Z

This pull request has gone a while without any activity. Tagging for triage help: @mosabua

Praveen2112 · 2024-12-26T09:13:29Z

Sorry for the delay. I'll take a look this week.

Praveen2112 · 2024-12-27T13:38:28Z

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

            analysis.setScope(node, queryScope);
+            ImmutableList<OutputColumn> outputColumns = outputColumnsBuilder.build();


Praveen2112 · 2024-12-27T13:42:50Z

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

+                        }
+                    }
+                }
+                analysis.setUpdateTarget(version, qualifiedName, Optional.empty(), Optional.of(outputColumns));


I'm not sure if using updateTarget is a right approach here as it is used for a different set of operations and we can't bind it to the first table we encounter as a part of SELECT query

I did think about this and tried to first add a dedicated target but it required quite a bit of plumbing and changes creeped in across many classes and ultimately the EventListener as well; hence the snippet in the description of the PR. So I wasn't sure if we want to go down a major change.
I do agree it's somewhat awkward to tie a select to a first table but ultimately binding a select to table is kinda irrelevant conceptually and should be just ignored. I am happy to change course on this as well.

Praveen2112 · 2024-12-27T13:43:37Z

core/trino-main/src/main/java/io/trino/sql/analyzer/StatementAnalyzer.java

+            if (!outputColumns.isEmpty()) {
+                QualifiedObjectName qualifiedName = new QualifiedObjectName("", "", "");
+                CatalogHandle.CatalogVersion version = new CatalogHandle.CatalogVersion("1");
+                if (node.getQueryBody() instanceof QuerySpecification querySpecification) {


This check is a bit brittle as we capture them only for simple queries and not for queries which has WITH clause in the begining.

Praveen2112 · 2024-12-27T13:46:54Z

testing/trino-tests/src/test/java/io/trino/execution/TestEventListenerWithSplits.java

@@ -122,7 +123,10 @@ public void testSplitsForNormalQuery()
        QueryCompletedEvent queryCompletedEvent = queryEvents.getQueryCompletedEvent();
        assertThat(queryCompletedEvent.getContext().getResourceGroupId().isPresent()).isTrue();
        assertThat(queryCompletedEvent.getContext().getResourceGroupId().get()).isEqualTo(createResourceGroupId("global", "user-user"));
-        assertThat(queryCompletedEvent.getIoMetadata().getOutput()).isEqualTo(Optional.empty());
+        Optional<QueryOutputMetadata> output = queryCompletedEvent.getIoMetadata().getOutput();


Instead of capturing this event for SELECT in QueryOutputMetadata ? Can we add additional field in QueryIOMetadata where there could be a Optional<List<OutputColumnMetadata>> which could be set only for SELECT statements in this way the logic could be simplified.

@martint - Is QueryIOMetadata a right place to add them or is there a better place we could store them ?

I did take a stab at storing these info separately from output and as I mentioned here it (changes) quickly leaks to many places (QueryInfo.java, QueryStateMachine.java, QueryMonitor.java, QueryCompletedEvent.java, ...) to plumb this info from analysis all the way to to event listener.
So basically my idea was to avoid that and reuse update target for carrying the same info. In any case, I put a gist of the actual diff if I were to hold this in a separate variable in the analysis, CompletedEvent, and QueryInfo. I am happy to update this PR to reflect that if you folks prefer that.
CC: @martint @Praveen2112

behrooz-stripe marked this pull request as ready for review September 10, 2024 00:10

behrooz-stripe marked this pull request as draft September 10, 2024 00:11

behrooz-stripe marked this pull request as ready for review September 16, 2024 19:25

behrooz-stripe force-pushed the master branch from c0c6780 to fd23000 Compare September 17, 2024 17:01

behrooz-stripe force-pushed the master branch from 1e2edd3 to eecace9 Compare September 17, 2024 17:22

behrooz-stripe force-pushed the master branch from eecace9 to e584f30 Compare September 18, 2024 21:29

cla-bot bot added the cla-signed label Sep 18, 2024

add column lineage info for all queries to output target

92177c0

behrooz-stripe force-pushed the master branch from e584f30 to 92177c0 Compare September 23, 2024 18:41

behrooz-stripe assigned electrum and wendigo Oct 7, 2024

wendigo removed their assignment Oct 7, 2024

hashhar requested a review from Praveen2112 October 8, 2024 10:02

github-actions bot added the stale label Dec 25, 2024

github-actions bot removed the stale label Dec 26, 2024

Praveen2112 reviewed Dec 27, 2024

View reviewed changes

		analysis.setScope(node, queryScope);
		ImmutableList<OutputColumn> outputColumns = outputColumnsBuilder.build();

Include Column Lineage Info for All Queries #23322

Are you sure you want to change the base?

Include Column Lineage Info for All Queries #23322

Conversation

behrooz-stripe commented Sep 6, 2024 • edited Loading

Description

Additional context and related issues

Release notes

cla-bot bot commented Sep 6, 2024

cla-bot bot commented Sep 6, 2024

cla-bot bot commented Sep 6, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 10, 2024

cla-bot bot commented Sep 12, 2024

cla-bot bot commented Sep 12, 2024

cla-bot bot commented Sep 13, 2024

cla-bot bot commented Sep 13, 2024

cla-bot bot commented Sep 13, 2024

cla-bot bot commented Sep 13, 2024

cla-bot bot commented Sep 13, 2024

behrooz-stripe commented Sep 16, 2024

mosabua commented Sep 16, 2024

behrooz-stripe commented Sep 16, 2024

wendigo commented Sep 17, 2024

cla-bot bot commented Sep 17, 2024

cla-bot bot commented Sep 17, 2024

behrooz-stripe commented Sep 17, 2024

wendigo commented Sep 17, 2024

cla-bot bot commented Sep 17, 2024

behrooz-stripe commented Sep 17, 2024

behrooz-stripe commented Sep 17, 2024

behrooz-stripe commented Sep 17, 2024

behrooz-stripe commented Sep 23, 2024

wendigo commented Oct 7, 2024

behrooz-stripe commented Oct 7, 2024

patstevens4 commented Oct 28, 2024

hashhar commented Nov 18, 2024

behrooz-stripe commented Dec 3, 2024

github-actions bot commented Dec 25, 2024

Praveen2112 commented Dec 26, 2024

Praveen2112 Dec 27, 2024

Choose a reason for hiding this comment

Praveen2112 Dec 27, 2024

Choose a reason for hiding this comment

behrooz-stripe Jan 6, 2025

Choose a reason for hiding this comment

Praveen2112 Dec 27, 2024

Choose a reason for hiding this comment

Praveen2112 Dec 27, 2024

Choose a reason for hiding this comment

behrooz-stripe Jan 8, 2025 • edited Loading

Choose a reason for hiding this comment

behrooz-stripe commented Sep 6, 2024 •

edited

Loading

behrooz-stripe Jan 8, 2025 •

edited

Loading