Skip to content

Commit

Permalink
Fix pytest from working outside breeze (apache#43082)
Browse files Browse the repository at this point in the history
This was missed in apache#42985 . Without this `airflow.providers.__path__` had 2 registered paths:

```
['/Users/kaxilnaik/Documents/GitHub/astronomer/airflow/providers/src/airflow/providers',
 '/Users/kaxilnaik/Documents/GitHub/astronomer/airflow/providers']
```

This prevents the tests from running outside of breeze and we get the following error:

```
ERROR tests/core/test_settings.py::test_usage_data_collection_disabled[true-True-True] - airflow.exceptions.AirflowConfigException: ("The provider apache-airflow-providers-src-airflow-providers-amazon is attempting to contribute configuration section aws that has already been added before. The source of it: apache-airflow-providers-amazon. This is forbidden. A provider can only add new sections. It cannot contribute options to existing sections or override other provider's configuration.", <class 'UserWarning'>)
```

We get this error because the *Providers Manager* uses `airflow.providers.__path__` to register providers. Because we have 2 paths, it registers the same provider twice leading two the above error.

https://github.com/apache/airflow/blob/75b22940ac4d36c31380669da2aa32fe46d70d32/airflow/providers_manager.py#L662

Example registration:
```
('apache-airflow-providers-src-airflow-providers-yandex',
  {'yandex': {'description': 'This section contains settings for Yandex Cloud ', ...

('apache-airflow-providers-yandex',
  {'yandex': {'description': 'This section contains settings for Yandex Cloud '
```

This wasn't a problem in breeze as it sets `AIRFLOW_SOURCES` env var in Dockerfile
https://github.com/apache/airflow/blob/75b22940ac4d36c31380669da2aa32fe46d70d32/scripts/docker/entrypoint_ci.sh#L24
  • Loading branch information
kaxil authored and PaulKobow7536 committed Oct 24, 2024
1 parent b131204 commit 5574944
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion tests_common/pytest_plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@

_airflow_sources = os.getenv("AIRFLOW_SOURCES", None)
AIRFLOW_SOURCES_ROOT_DIR = (
Path(_airflow_sources) if _airflow_sources else Path(__file__).parents[2]
Path(_airflow_sources) if _airflow_sources else Path(__file__).parents[1]
).resolve()
AIRFLOW_TESTS_DIR = AIRFLOW_SOURCES_ROOT_DIR / "tests"

Expand Down Expand Up @@ -376,6 +376,8 @@ def pytest_configure(config: pytest.Config) -> None:
if path == desired:
break
else:
# This "desired" path should be the Airflow source directory (repo root)
assert (AIRFLOW_SOURCES_ROOT_DIR / ".asf.yaml").exists(), f"Path {desired} is not Airflow root"
sys.path.append(desired)

if (backend := config.getoption("backend", default=None)) and backend not in SUPPORTED_DB_BACKENDS:
Expand Down

0 comments on commit 5574944

Please sign in to comment.