openlineage, snowflake: add OpenLineage support for Snowflake #31696
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
JDarDagran
requested review from
potiuk,
ashb,
jedcunningham,
eladkal and
mik-laj
as code owners
boring-cyborg
bot
added
area:dev-tools
area:providers
provider:common-sql
provider:openlineage
sunank200
reviewed
Jun 13, 2023
| authority = f"{parsed.hostname}:{parsed.port}" | ||
| return authority | ||
|
|
||
| def get_default_schema(self): |
| """ | ||
| return self.__schema or "public" | ||
|
|
||
| def get_database_specific_lineage(self, task_instance): |
There was a problem hiding this comment.
isn't there specific lineage in case of snowflake - eg. query run time
There was a problem hiding this comment.
For now I'm trying to reflect what's currently sent from openlineage-airflow package.
I might add some more information but I'd really appreciate what would be useful and available to retrieve.
There was a problem hiding this comment.
On the other hand - https://docs.snowflake.com/en/sql-reference/account-usage/query_history#usage-notes: latency for QUERY_HISTORY is up to 45 minutes so it looks like it's not sustainable to always get_ query run time
JDarDagran
force-pushed
the
aip-53-snowflake
branch
2 times, most recently
from
edb89d4
to
3c4844f
Compare
|
One failing test looking rather flaky. |
Signed-off-by: Jakub Dardzinski <[email protected]>
Signed-off-by: Jakub Dardzinski <[email protected]>
JDarDagran
force-pushed
the
aip-53-snowflake
branch
from
3c4844f
to
dae1ba3
Compare
mobuchowski
approved these changes
Jul 21, 2023
69 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds OpenLineage support for SnowflakeOperator.
depends on: #31398