Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#…

…15048)

* Upgrades moto to newer version (~=2.0)

According to getmoto/moto#3535 (comment)
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.

At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.

Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).

* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix

With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: #14994
(cherry picked from commit ec962b0)

Dockerfile

@@ -276,7 +276,8 @@ ENV INSTALL_FROM_PYPI=${INSTALL_FROM_PYPI}
 # * chardet<4 - required to keep snowflake happy
 # * urllib3 - required to keep boto3 happy
 # * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
 
 WORKDIR /opt/airflow
 

Dockerfile.ci

@@ -348,7 +348,8 @@ COPY airflow/__init__.py ${AIRFLOW_SOURCES}/airflow/__init__.py
 # * pyOpenSSL: required by snowflake provider https://github.com/snowflakedb/snowflake-connector-python/blob/v2.3.6/setup.py#L201
 # * urllib3<1.26: Required to keep boto3 happy
 # * pyjwt<2.0.0: flask-jwt-extended requires it
-ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0"
+# * dill<0.3.3 required by apache-beam
+ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="chardet<4 lazy-object-proxy<1.5.0 pyOpenSSL<20.0.0 urllib3<1.26 pyjwt<2.0.0 dill<0.3.3"
 ENV EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS=${EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS}
 
 ARG CONTINUE_ON_PIP_CHECK_FAILURE="false"

setup.py

@@ -742,8 +742,11 @@ def add_extras_for_all_deprecated_aliases() -> None:
 # This can be removed as soon as we get non-conflicting
 # requirements for the apache-beam as well.
 #
-# Currently Apache Beam has very narrow and old dependencies for 'dill' and 'mock' packages which
-# are required by our tests (but only for tests).
+# Currently Apache Beam has very narrow and old dependencies for 'mock' package which
+# are required only for our tests.
+# once https://github.com/apache/beam/pull/14328 is solved and new version of apache-beam is released
+# we will be able to remove this exclusion and get rid of `install_remaining_dependencies`
+# function in `scripts/in_container`.
 #
 PACKAGES_EXCLUDED_FOR_CI = [
     'apache-beam',