Skip to content
This repository has been archived by the owner on Sep 4, 2024. It is now read-only.

Deprecate the provider and proxy to upstream Airflow provider #84

Merged
merged 5 commits into from
Aug 8, 2024

Conversation

pankajkoti
Copy link
Collaborator

@pankajkoti pankajkoti commented Aug 5, 2024

As part of Astronomer's internal plans and decisions, we've decided to contribute the existing functionality provided by the operators and plugins in this repository to the official Apache Airflow Databricks provider. To achieve this, we submitted the following PRs to the Airflow provider:

  1. Add DatabricksNotebookOperator apache/airflow#39178
  2. Add DatabricksWorkflowTaskGroup apache/airflow#39771
  3. Add DatabricksTaskOperator apache/airflow#40013
  4. Add DatabricksWorkflowPlugin apache/airflow#40724
  5. add deferrable support to DatabricksNotebookOperator apache/airflow#39295

All functionality has now been contributed to the Airflow Databricks provider, and ongoing support will be maintained there. As a result, we're deprecating the operators and plugins in this repository. Users are encouraged to transition to the official Apache Airflow Databricks provider as soon as possible. The migration process is straightforward—simply update the import path to point to the Airflow provider and ensure that you install apache-airflow-providers-databricks>=6.8.0, which includes all the contributions mentioned above.

closes: https://github.com/astronomer/issues-airflow/issues/715

pyproject.toml Outdated Show resolved Hide resolved
@pankajkoti pankajkoti marked this pull request as ready for review August 5, 2024 09:23
@pankajkoti
Copy link
Collaborator Author

I have tested the proxying by running all example DAGs successfully along with testing plugin functionality by calling repair on tasks.

Screenshot 2024-08-05 at 2 44 42 PM Screenshot 2024-08-05 at 2 44 48 PM Screenshot 2024-08-05 at 2 45 09 PM Screenshot 2024-08-05 at 2 44 16 PM

.github/workflows/ci.yml Outdated Show resolved Hide resolved
@codecov-commenter
Copy link

codecov-commenter commented Aug 5, 2024

Codecov Report

Attention: Patch coverage is 94.44444% with 2 lines in your changes missing coverage. Please review.

Project coverage is 92.45%. Comparing base (8b27e03) to head (abc416f).
Report is 1 commits behind head on main.

Files Patch % Lines
src/astro_databricks/plugins/plugin.py 90.47% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main      #84       +/-   ##
===========================================
+ Coverage   77.51%   92.45%   +14.94%     
===========================================
  Files           6        6               
  Lines         627       53      -574     
  Branches       89        0       -89     
===========================================
- Hits          486       49      -437     
+ Misses        113        4      -109     
+ Partials       28        0       -28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link

@pankajastro pankajastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

README.md Show resolved Hide resolved
Copy link
Collaborator

@tatiana tatiana left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant work, @pankajkoti , it looks great - can't wait to see this released!

@pankajkoti pankajkoti merged commit 3c858a4 into main Aug 8, 2024
13 checks passed
@pankajkoti pankajkoti deleted the deprecate-provider branch August 8, 2024 09:29
@pankajkoti pankajkoti mentioned this pull request Aug 8, 2024
pankajkoti added a commit that referenced this pull request Aug 8, 2024
Deprecations

Deprecate the provider and proxy instantiations to upstream official Apache Airflow Databricks provider (PR Deprecate the provider and proxy to upstream Airflow provider #84 )
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants