Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix

With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: apache#14994

Dockerfile

@@ -276,7 +276,8 @@ ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}
 # * chardet<4 - required to keep snowflake happy
 # * urllib3 - required to keep boto3 happy
 # * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
 
 WORKDIR /opt/airflow
 

Dockerfile.ci

@@ -348,7 +348,8 @@ COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py
 # * pyOpenSSL: required by snowflake provider https://github.com/snowflakedb/snowflake-connector-python/blob/v2.3.6/setup.py#L201
 # * urllib3<1.26: Required to keep boto3 happy
 # * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
 ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
 
 ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"

setup.py

@@ -743,8 +743,11 @@ def add_extras_for_all_deprecated_aliases() -> None:
 # This can be removed as soon as we get non-conflicting
 # requirements for the apache-beam as well.
 #
-# Currently Apache Beam has very narrow and old dependencies for 'dill' and 'mock' packages which
-# are required by our tests (but only for tests).
+# Currently Apache Beam has very narrow and old dependencies for 'mock' package which
+# are required only for our tests.
+# once https://github.com/apache/beam/pull/14328 is solved and new version of apache-beam is released
+# we will be able to remove this exclusion and get rid of `install_remaining_dependencies`
+# function in `scripts/in_container`.
 #
 PACKAGES_EXCLUDED_FOR_CI = [
     'apache-beam',