Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenLineage] Use prefixes instead of file paths for datasets in GCSToGCSOperator #39058

Merged
merged 1 commit into from
Apr 18, 2024

Conversation

kacpermuda
Copy link
Contributor

@kacpermuda kacpermuda commented Apr 16, 2024

Currently we are including all copied files as datasets which can lead to increasing the size of the event and make matching datasets between jobs harder.

With that change, we are using prefixes from the user as dataset names and not full file paths. This way, user can easily control the size of the event and also ensure proper matching, when the same two prefixes are passed to different operators. I am also removing the list of files that was saved for the purpose of lineage datasets, introduced in #31350 .

When reviewing, please take a look at test cases to see how the code will behave now.

Also, a small adjustment in gcs_to_bigquery was made - just an optimization with no logic change.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Apr 16, 2024
@kacpermuda kacpermuda force-pushed the ol-fix-gcs-to-gcs branch 2 times, most recently from a33080c to 9c99645 Compare April 16, 2024 10:35
@mobuchowski mobuchowski self-requested a review April 17, 2024 09:56
@mobuchowski
Copy link
Contributor

@kacpermuda looks good to me, but can you just change commit title? It does not reflect the content.

@kacpermuda kacpermuda changed the title [OpenLineage] Repair datasets in gcs_to_gcs operator [OpenLineage] Use prefixes instead of file paths for datasets in GCSToGCSOperator Apr 18, 2024
@kacpermuda
Copy link
Contributor Author

@mobuchowski Done, obvious typo in the commit message, thanks for catching that !

@kacpermuda
Copy link
Contributor Author

Not sure why the warnings suddenly fail the tests. I am testing a deprecated issue, yes, but it still should be tested 😄 Should i catch and ignore all warnings?

@mobuchowski mobuchowski merged commit 927e364 into apache:main Apr 18, 2024
41 checks passed
@kacpermuda kacpermuda deleted the ol-fix-gcs-to-gcs branch April 18, 2024 14:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants