Add DatabricksNotebookOperator
#39178
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
pankajkoti
force-pushed
the
contribute-databricks-notebook-operator
branch
2 times, most recently
from
a98d220
to
d5fb847
Compare
tatiana
reviewed
Apr 23, 2024
There was a problem hiding this comment.
It's looking very good, @pankajkoti , thanks for contributing this!
pankajkoti
changed the title
pankajkoti
commented
Apr 23, 2024
pankajkoti
force-pushed
the
contribute-databricks-notebook-operator
branch
from
fb7b1a3
to
bd08d9d
Compare
tatiana
approved these changes
Apr 23, 2024
sunank200
approved these changes
Apr 24, 2024
eladkal
changed the title
DatabricksNotebookOperator
Lee-W
reviewed
Apr 25, 2024
Co-authored-by: Tatiana Al-Chueyr <[email protected]>
Co-authored-by: Wei Lee <[email protected]>
pankajkoti
force-pushed
the
contribute-databricks-notebook-operator
branch
from
be84ec0
to
20dacc7
Compare
Lee-W
approved these changes
Apr 26, 2024
pankajkoti
pushed a commit
that referenced
this pull request
May 14, 2024
related: #39178 This PR intends to make DatabricksNotebookOperator deferrable
pankajkoti
added a commit
that referenced
this pull request
May 30, 2024
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being #39178. The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``. --------- Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: Tatiana Al-Chueyr <[email protected]> Co-authored-by: Wei Lee <[email protected]>
fdemiane
pushed a commit
to fdemiane/airflow
that referenced
this pull request
Jun 6, 2024
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being apache#39178. The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``. --------- Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: Tatiana Al-Chueyr <[email protected]> Co-authored-by: Wei Lee <[email protected]>
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
related: apache#39178 This PR intends to make DatabricksNotebookOperator deferrable
romsharon98
pushed a commit
to romsharon98/airflow
that referenced
this pull request
Jul 26, 2024
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being apache#39178. The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``. --------- Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: Tatiana Al-Chueyr <[email protected]> Co-authored-by: Wei Lee <[email protected]>
pankajkoti
added a commit
to astronomer/astro-provider-databricks
that referenced
this pull request
Aug 8, 2024
As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider: 1. apache/airflow#39178 2. apache/airflow#39771 3. apache/airflow#40013 4. apache/airflow#40724 5. apache/airflow#39295 All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install `apache-airflow-providers-databricks>=6.8.0`, which includes all the contributions mentioned above. closes: astronomer/issues-airflow#715
kosteev
pushed a commit
to GoogleCloudPlatform/composer-airflow
that referenced
this pull request
Nov 9, 2024
This pull request introduces the [DatabricksWorkflowTaskGroup](https://github.com/astronomer/astro-provider-databricks/blob/main/src/astro_databricks/operators/workflow.py#L226) to the Airflow Databricks provider from the [astro-provider-databricks](https://github.com/astronomer/astro-provider-databricks/tree/main) repository. It marks another pull request aimed at contributing operators and features from that repository into the Airflow Databricks provider, the previous PR being apache/airflow#39178. The task group launches a [Databricks Workflow](https://docs.databricks.com/en/workflows/index.html) and runs the notebook jobs from within it, resulting in a [75% cost reduction](https://www.databricks.com/product/pricing) ($0.40/DBU for all-purpose compute, $0.07/DBU for Jobs compute) when compared to executing ``DatabricksNotebookOperator`` outside of ``DatabricksWorkflowTaskGroup``. --------- Co-authored-by: Daniel Imberman <[email protected]> Co-authored-by: Tatiana Al-Chueyr <[email protected]> Co-authored-by: Wei Lee <[email protected]> GitOrigin-RevId: 2ecf7fa07d6d681c73ae4831801f9d98db298d89
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces the DatabricksNotebookOperator
to the Airflow Databricks provider from the astro-provider-databricks
repository.
It marks one of the initial pull requests aimed at contributing
operators and features from that repository into the Airflow
Databricks provider.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.