[airflow]: Import modules that has been moved to airflow providers (AIR303)

[Lee-W]

Summary

Many core Airflow features have been deprecated and moved to Airflow Providers since users might need to install an additional package (e.g., apache-airflow-provider-fab==1.0.0); a separate rule (AIR303) is created for this.

As some of the changes only relate to the module/package moved, instead of listing out all the functions, variables, and classes in a module or a package, it warns the user to import from the new path instead of the specific name.

The following is the ones that has been moved to apache-airflow-provider-fab==1.0.0

• module moved
  • airflow.api.auth.backend.basic_auth → airflow.providers.fab.auth_manager.api.auth.backend.basic_auth
  • airflow.api.auth.backend.kerberos_auth → airflow.providers.fab.auth_manager.api.auth.backend.kerberos_auth
  • airflow.auth.managers.fab.api.auth.backend.kerberos_auth → airflow.providers.fab.auth_manager.api.auth.backend.kerberos_auth
  • airflow.auth.managers.fab.security_manager.override → airflow.providers.fab.auth_manager.security_manager.override
• classes (e.g., functions, classes) moved
  • airflow.www.security.FabAirflowSecurityManagerOverride → airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride
  • airflow.auth.managers.fab.fab_auth_manager.FabAuthManager → airflow.providers.fab.auth_manager.security_manager.FabAuthManager

#14764

How to contribute to AIR303

Browse the airflow / ruff migration guideline

How to create a Ruff lint rule for Airflow 2-to-3 migrations

Setup local development environment

cargo install cargo-insta

uv tool install pre-commit
pre-commit install

cargo install cargo-nextest --locked

Contributing to Ruff

Check the existing rules from airflow issue

All the ruff rules are listed down in Airflow 2 to 3 auto migration rules - ruff #44556

Go to AIR303: moved to provider in the issue description and you'll find rules like

* `airflow.www.security.FabAirflowSecurityManagerOverride` → `airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride` (from https://github.com/apache/airflow/pull/41758)

It means airflow.www.security.FabAirflowSecurityManagerOverride has been moved to airflow provider and the import should be changed to airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride. This changed is made in apache/airflow#41758

Some of the rules might not be correct, so checking the pull request again is suggested.

Add a new rule

Go to crates/ruff_linter/src/rules/airflow/rules/moved_to_provider_in_3.rs

In function moved_to_provider, add code section like the following

                ["airflow", "www", "security", "FabAirflowSecurityManagerOverride"] => Some((
                    qualname.to_string(),
                    Replacement::ProviderName {
                        name: "airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride",
                        provider: "fab",
                        version: "1.0.0"
                    },
                )),

• "airflow", "www", "security", "FabAirflowSecurityManagerOverride": the deprecated import path
• name is the new name that should be used.
• provider is which provider this name has been moved to.
• version is the first version of the provider that has been moved to.

For rules like

* [ ] `airflow.api.auth.backend.basic_auth` → `airflow.providers.fab.auth_manager.api.auth.backend.basic_auth` (from https://github.com/apache/airflow/pull/41663)

The whole module/pacakge has been moved to the provider. Thus, we'll need to warn the users when something is imported from the old path instead.

                ["airflow", "api", "auth", "backend", "basic_auth", ..] => Some((
                    qualname.to_string(),
                    Replacement::ImportPathMoved{
                        original_path: "airflow.api.auth.backend.basic_auth",
                        new_path: "airflow.providers.fab.auth_manager.api.auth.backend.basic_auth",
                        provider:"fab",
                        version: "1.0.0"
                },
                )),

• "airflow", "api", "auth", "backend", "basic_auth", ..: anything under this import path
• original path
• new path: new import path in the provider
• provider is which provider this name has been moved to.
• version is the first version of the provider that has been moved to.

Add test case

Add a case that violate this rule to crates/ruff_linter/resources/test/fixtures/airflow/AIR303.py

Test and generate snapshop to review

cargo insta test
cargo insta review

Now you're ready to create a pull request

Test Plan

A test fixture has been included for the rule.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

[Lee-W]

@MichaReiser would love to know whether the overall idea of this PR makes sense if you have some time. I plan on writing a guideline for folks from the airflow community to contribute to AIR303 more easily. 🙂

[dhruvmanila]

Is it correct that the entire module itself has been moved to being used as a provider without changing any of the public symbols available in that module? i.e., are the symbols that were available in the old module the same as that in the provider module? If so, we could consider the rule to work on the module instead of individual symbols from the module and suggest to use the provider instead. That way we'll only emit a single diagnostic for the entire module which will be the import statement.

[Lee-W]

Is it correct that the entire module itself has been moved to being used as a provider without changing any of the public symbols available in that module? i.e., are the symbols that were available in the old module the same as that in the provider module?

Yep, that's what I think for some of the modules, but I'll confirm it again.

If so, we could consider the rule to work on the module instead of individual symbols from the module and suggest to use the provider instead. That way we'll only emit a single diagnostic for the entire module which will be the import statement.

Yep, some of the modules can be done this way, but others might not, so that why I'd like to keep the individual symbols check as well

[dhruvmanila]

Yep, some of the modules can be done this way, but others might not, so that why I'd like to keep the individual symbols check as well

That's sounds reasonable.

[MichaReiser]

writing a guideline for folks from the airflow community to contribute to AIR303 more easily.

This sounds great. I'd appreciate your with reviewing and coordinating the PRs as you have the most knowledge about what the rules should cover.

[Lee-W]

writing a guideline for folks from the airflow community to contribute to AIR303 more easily.

This sounds great. I'd appreciate your with reviewing and coordinating the PRs as you have the most knowledge about what the rules should cover.

Will do 👍