Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

openlineage, gcs: add openlineage methods for GcsToGcsOperator #31350

Merged
merged 1 commit into from
Jul 27, 2023
Merged

openlineage, gcs: add openlineage methods for GcsToGcsOperator #31350

merged 1 commit into from
Jul 27, 2023

Conversation

mobuchowski
Copy link
Contributor

@mobuchowski mobuchowski commented May 17, 2023 •
edited
Loading

This PR adds OpenLineage support for GcsToGcsOperator.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels May 17, 2023
sunank200
sunank200 reviewed May 18, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should add documentation around this along with the following changes

airflow/providers/google/cloud/transfers/gcs_to_gcs.py Outdated Show resolved Hide resolved
airflow/providers/google/cloud/transfers/gcs_to_gcs.py
return None

return OperatorLineage(
inputs=[
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about DataSource dataset facet as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#dataset-facets?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not required @mobuchowski ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's required here

@mobuchowski
Copy link
Contributor Author

@sunank200

Add run facet which captures the error as per https://github.com/OpenLineage/OpenLineage/blob/main/spec/OpenLineage.md#run-facets
Add test case for error scenario where runfacet is populated

In general case it's hard to capture errors now, since on_task_instance_failed does not pass actual error message.
We could store it somewhere on task_instance maybe.

@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 2 times, most recently from 9af4382 to edf62f9 Compare May 25, 2023 12:24
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 2 times, most recently from 65afad8 to cf86fb7 Compare June 15, 2023 15:02
@pankajkoti pankajkoti removed their request for review July 12, 2023 12:22
@mobuchowski mobuchowski force-pushed the openlineage-gcs-operator branch 4 times, most recently from 26d29bb to 0b83011 Compare July 24, 2023 11:52
sunank200
sunank200 approved these changes Jul 24, 2023
View reviewed changes
airflow/providers/google/cloud/transfers/gcs_to_gcs.py
return None

return OperatorLineage(
inputs=[
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this not required @mobuchowski ?

sunank200
sunank200 approved these changes Jul 24, 2023
View reviewed changes
Copy link
Collaborator

@sunank200 sunank200 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once test passes it should be good to be merged.

@eladkal eladkal merged commit b733667 into apache:main Jul 27, 2023
@eladkal eladkal mentioned this pull request Jul 29, 2023
Closed
69 tasks
@kacpermuda kacpermuda mentioned this pull request Apr 16, 2024
Merged
@eladkal eladkal mentioned this pull request May 1, 2024
Closed
@eladkal eladkal mentioned this pull request May 12, 2024
Closed
66 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eladkal eladkal eladkal approved these changes

@ahidalgob ahidalgob ahidalgob approved these changes

@sunank200 sunank200 sunank200 approved these changes

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@phanikumv phanikumv Awaiting requested review from phanikumv

@pankajastro pankajastro Awaiting requested review from pankajastro

Assignees
No one assigned
Labels
area:providers provider:google Google (including GCP) related issues provider:openlineage AIP-53
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mobuchowski @sunank200 @ahidalgob @eladkal