Fix dependencies for celery and opentelemetry for Python 3.8 (#33579)

We used to have problems with `pip` backtracking when we relaxed
too much open-telemetry dependencies. It turned out that the
backtracting was only happening on Python 3.8 and that it was
ultimately caused by conflict between importlib_metadata between
Airflow and newer versions of opentelemetry (we had <5 for Python
3.8, they had >6 for all versions. The reason for limiting it in
Airflow was Celery that was not working well with importlib 5.

Since Celery 5.3 solved the problems (released 6th of June) we can
now relax the importlib_metadata limit and set Celery to version >=
5.3.0) which nicely resolves the conflict and there is no more
backtracking when trying to install newer versions of opentelemetry
for Python 3.8.

Fixes: #33577

airflow/providers/celery/provider.yaml

@@ -45,7 +45,7 @@ dependencies:
   # Uses Celery for CeleryExecutor, and we also know that Kubernetes Python client follows SemVer
   # (https://docs.celeryq.dev/en/stable/contributing.html?highlight=semver#versions).
   # Make sure that the limit here is synchronized with [celery] extra in the airflow core
-  - celery>=5.2.3,<6
+  - celery>=5.3.0,<6
   - flower>=1.0.0
   - google-re2>=1.0
 

dev/breeze/src/airflow_breeze/utils/path_utils.py

@@ -87,7 +87,7 @@ def get_package_setup_metadata_hash() -> str:
     try:
         from importlib.metadata import distribution  # type: ignore[attr-defined]
     except ImportError:
-        from importlib_metadata import distribution  # type: ignore[no-redef]
+        from importlib_metadata import distribution  # type: ignore[no-redef, assignment]
 
     prefix = "Package config hash: "
 

generated/provider_dependencies.json

@@ -245,7 +245,7 @@
   "celery": {
     "deps": [
       "apache-airflow>=2.4.0",
-      "celery>=5.2.3,<6",
+      "celery>=5.3.0,<6",
       "flower>=1.0.0",
       "google-re2>=1.0"
     ],

setup.cfg

@@ -102,10 +102,7 @@ install_requires =
     graphviz>=0.12
     gunicorn>=20.1.0
     httpx
-    # Importlib-metadata 5 is breaking Celery import due to regression it introduced
-    # This was tracked and fixed in https://github.com/celery/celery/pull/7785 but it is not released yet
-    # We can remove the < 5.0.0 limitation when Celery 5.3.0 gets released and we bump celery to >= 5.3.0
-    importlib_metadata>=1.7,<5.0.0;python_version<"3.9"
+    importlib_metadata>=1.7;python_version<"3.9"
     importlib_resources>=5.2;python_version<"3.9"
     itsdangerous>=2.0
     jinja2>=3.0.0
@@ -118,8 +115,7 @@ install_requires =
     markupsafe>=1.1.1
     marshmallow-oneofschema>=2.0.1
     mdit-py-plugins>=0.3.0
-    # Pip can not find a version that satisfies constraints if opentelemetry-api is not pinned.
-    opentelemetry-api==1.15.0
+    opentelemetry-api>=1.15.0
     opentelemetry-exporter-otlp
     packaging>=14.0
     pathspec>=0.9.0

setup.py

@@ -281,7 +281,7 @@ def write_version(filename: str = str(AIRFLOW_SOURCES_ROOT / "airflow" / "git_ve
     # limiting minimum airflow version supported in celery provider due to the
     # potential breaking changes in Airflow Core as well (celery is added as extra, so Airflow
     # core is not hard-limited via install-requires, only by extra).
-    "celery>=5.2.3,<6"
+    "celery>=5.3.0,<6"
 ]
 cgroups = [
     # Cgroupspy 0.2.2 added Python 3.10 compatibility