Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove the XCOM pull and push for Open lineage #1069

Open
sunank200 opened this issue Oct 14, 2022 · 0 comments
Open

Remove the XCOM pull and push for Open lineage #1069

sunank200 opened this issue Oct 14, 2022 · 0 comments
Labels
product/python-sdk Label describing products

Comments

@sunank200
Copy link
Contributor

Describe the bug
Currently we are pushing the values to xcom to get the values in extract_on_complete for open lineage. This is a temporary hack till the issue is not fixed on airflow side apache/airflow#21157

This work after changing the following line from taskinstance.py:
session.merge(self)
to
session.merge(self).task = self.task
With this, I was able to get it to work
task_instance.task.sql
The only change then I have in python-sdk/src/astro/lineage/extractor.py is :
Before:
self.log.debug(self.operator.sql)
After:
self.log.debug(task_instance.task.sql)
Without that change, it didn’t work because OpenLineage saves the “task” before running (before calling the op.execute method but after TI is set to Running state) the task in memory - this line in on_task_instance_running

Expected behavior
Remove xcom push and pull on operator

@pankajastro pankajastro added the product/python-sdk Label describing products label Oct 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
product/python-sdk Label describing products
Projects
None yet
Development

No branches or pull requests

2 participants