Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement OpenLineage extractor for Redshift async operators #462

Closed
2 tasks done
pankajkoti opened this issue Jun 22, 2022 · 12 comments · Fixed by #561
Closed
2 tasks done

Implement OpenLineage extractor for Redshift async operators #462

pankajkoti opened this issue Jun 22, 2022 · 12 comments · Fixed by #561
Assignees
Labels
pri/medium Medium priority

Comments

@pankajkoti
Copy link
Collaborator

pankajkoti commented Jun 22, 2022

Implement OpenLineage extractor for Redshift async operators

  • RedshiftSQLOperatorAsync
  • RedshiftDataOperatorAsync
@pankajkoti pankajkoti added the pri/medium Medium priority label Jul 4, 2022
@rajaths010494
Copy link
Contributor

Broken plugin: [openlineage.airflow.plugin] Failed to import astronomer.providers.amazon.aws.extractors.redshift.RedshiftAsyncExtractor .
Update OL version to 0.10.0 and 0.11.0 and tried the same issue exists.
Connect with @pankajkoti and try to resolve this.

@pankajkoti
Copy link
Collaborator Author

Current issue is that with the latest Astro image and recent OpenLineage releases 0.9.0, 0.10.0 & 0.11.0 the plugin load is failing for Redshift Extractor. @rajaths010494 to connect with Maciej (GetInData team) and dig deeper into resolving this issue today.

@rajaths010494
Copy link
Contributor

Waiting for PR to be raised from Ol side so that i can review and implement the same in our extractor

@pankajkoti
Copy link
Collaborator Author

We can refer to this page: https://www.notion.so/astronomerio/Developing-OpenLineage-in-Astro-9f30a74c74354f168279c1205f4d89a6 to install local packages

@rajaths010494
Copy link
Contributor

Today have a connect with maciej for broken plugin issue.

@rajaths010494
Copy link
Contributor

Draft PR has been raised #561

@rajaths010494
Copy link
Contributor

Test cases are pending

@bharanidharan14
Copy link
Contributor

Expected to be completed EOD

@rajaths010494
Copy link
Contributor

New version of openlineage-airflow==0.12.0 has been released, will revamp unwanted code and then push the changes

@rajaths010494
Copy link
Contributor

rajaths010494 commented Aug 9, 2022

Have incorporated the new openlineage-airflow==0.12.0 code for redshift extractors.

@phanikumv
Copy link
Collaborator

are we ready with the PR for this? It is in draft state at the moment.

@rajaths010494
Copy link
Contributor

Ready for review and i have rebased

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pri/medium Medium priority
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants