Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dill version conflict fails CI #14994

Closed
uranusjr opened this issue Mar 24, 2021 · 3 comments · Fixed by #15048
Closed

dill version conflict fails CI #14994

uranusjr opened this issue Mar 24, 2021 · 3 comments · Fixed by #15048
Labels
area:CI Airflow's tests and continious integration kind:bug This is a clearly a bug

Comments

@uranusjr
Copy link
Member

uranusjr commented Mar 24, 2021 •
edited
Loading

Currently all PRs have their CI failing during the Check if CI image build is needed stage because pip check fails with

apache-beam 2.28.0 has requirement dill<0.3.2,>=0.3.1.1, but you have dill 0.3.3.

The source of the conflict is simple; Airflow itself declares dill>=0.2.2, <0.4, but apache-beam[gcp] (pulled in by extra) declares >=0.3.1.1,<0.3.2. The CI is currently using pip’s legacy resolver, which naively chooses in dill 0.3.3 when it seems the initial requirement in apache-airflow to cause the conflict.

The thing I’ve not figured out, however, is why the CI worked previously. Neither apache-airflow and apache-beam have changed this particular dependency declaration for quite a long time, and a similar conflict should have occured long ago.

As for a fix, the simplest solution would be to pin apache-airflow’s dill dependency to <0.3.2. Or switch to pip’s new resolver, which can correctly install apache-airflow[all] as of pip 21.0.1 from my test run.
@uranusjr uranusjr added the kind:bug This is a clearly a bug label Mar 24, 2021
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 24, 2021

Thanks for opening your first issue here! Be sure to follow the issue template!

@vikramkoka vikramkoka added the area:CI Airflow's tests and continious integration label Mar 25, 2021
@uranusjr uranusjr mentioned this issue Mar 25, 2021
Closed
@potiuk
Copy link
Member

potiuk commented Mar 26, 2021 •
edited
Loading

We are using constraints to limit the version of dill that is installed, however we know apache beam has a number of problems with dependencies. The reason we are not failing in CI is simple:

airflow/setup.py

Line 750 in fe2a70c

PACKAGES_EXCLUDED_FOR_CI = [ 

# Those packages are excluded because they break tests and they are not needed to run our test suite.
# This can be removed as soon as we get non-conflicting
# requirements for the apache-beam as well.
#
# Currently Apache Beam has very narrow and old dependencies for 'dill' and 'mock' packages which
# are required by our tests (but only for tests).
#
PACKAGES_EXCLUDED_FOR_CI = [
    'apache-beam',
]

The dill failing the build was a different reason entirely. This has been fixed in #14998. The problem was that a different bug caused the CI image to rebuild in an environment ( providers) where we do not care about pip check where we want to just check all providers importability. And this docker image build (which was unnecessary and harmful) was run in the way that did not exclude apache-beam and failed the pip check.

@potiuk potiuk closed this as completed Mar 26, 2021
@potiuk
Copy link
Member

potiuk commented Mar 27, 2021

The problem will be solved finaly in the next version of beam it seems - see apache/beam#14328.

In the meantime I will make a change that will make us ready to remove beam from this exclusion when new version is released without mock dependency.

@potiuk potiuk reopened this Mar 27, 2021
@potiuk potiuk mentioned this issue Mar 27, 2021
Merged
potiuk added a commit to potiuk/airflow that referenced this issue Mar 27, 2021
@potiuk
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix
8d42094 
With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: apache#14994
potiuk added a commit that referenced this issue Mar 27, 2021
@potiuk
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#…
ec962b0 
…15048)

* Upgrades moto to newer version (~=2.0)

According to getmoto/moto#3535 (comment)
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.

At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.

Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).

* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix

With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: #14994
potiuk added a commit that referenced this issue Mar 28, 2021
@potiuk
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#…
c25ae81 
…15048)

* Upgrades moto to newer version (~=2.0)

According to getmoto/moto#3535 (comment)
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.

At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.

Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).

* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix

With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: #14994
(cherry picked from commit ec962b0)
ashb pushed a commit that referenced this issue Apr 15, 2021
@potiuk @ashb
Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix (#…
588c1a1 
…15048)

* Upgrades moto to newer version (~=2.0)

According to getmoto/moto#3535 (comment)
1.3.17 version of moto with a fix to be compatible with mock> 4.0.3 is
not going to be released because of breaking changes. Therefore we need
to migrate to newer version of moto.

At the same time we can get rid of the old botocore limitation, which
was added apparently to handle some test errors. We are relying fully
on what boto3 depends on.

Upgrading dependencies also discovered that mysql tests need to
be fixed because upgraded version of dependencies cause some test
failure (those turned out to be badly written tests).

* Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix

With the upcoming apache-beam change where mock library will be
removed from install dependencies, we will be able to remove
`apache-beam` exclusion in our CI scripts. This will be a final
step of cleaning dependencies so that we have a truly
golden set of constraints that will allow to install airflow
and all community managed providers (we managed to fix all those
dependency issues for all packages but apache-beam).

The fix apache/beam#14328 when merged
and Apache Beam is released will allow us to migrate to the new
version and get rid of the CI exclusion for beam.

Closes: #14994
(cherry picked from commit ec962b0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area:CI Airflow's tests and continious integration kind:bug This is a clearly a bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Adds dill exclusion to Dockerfiles to accomodate upcoming beam fix potiuk/airflow
3 participants
@potiuk @uranusjr @vikramkoka