Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add instructions on how to avoid accidental airflow upgrade/downgrade #30813

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Apr 22, 2023

Some of our users raised issues that when extending the image, airflow suddenly started reporting problem with database versions and migration not aplied or out-of-sync. This almost always turns out to be a dependency conflict, that leads to automated downgrate or upgrade of installed airflow version. This is - obviously - undesired (you should be upgrading airflow consciously rather than accidentally). However there is no way to do it implicitly - pip might decide to upgrade or downgrade airflow as it sees fit. From the point of view - airflow is just one of the packages and has no special meaning.

The only way to "keep" airflow version is to specify it together with other requirements, pinned to the specific version. This PR updates our examples to do this and explains why airflow is added there.

There is - of course - another risk that the user will forget to update the version of airflow when they upgrade, however, sinc this is explicit action performed during image extension, it is much easier to diagnose and notice. We also warn the users that they should upgrade when airflow is upgraded.


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk requested a review from ashb as a code owner April 22, 2023 13:16
@boring-cyborg boring-cyborg bot added area:production-image Production image improvements and fixes kind:documentation labels Apr 22, 2023
@potiuk potiuk added this to the Airflow 2.6.0 milestone Apr 22, 2023
@potiuk
Copy link
Member Author

potiuk commented Apr 22, 2023

This is a doc-only change - updating our examples of docs extending and customising. I think it might be worthwile to cherry-pick it to 2.6.0, as it should help our users to avoid surprising upgrades/downgrade of airflow when they are adding conflicting dependencies. This was a recurring pattern in quite a number of Slack threads and github issues, the problem is that pip treats airflow as "just another python package" and when user installs conflicting dependencies, it will downgrade or upgrade airflow without asking (which is of course rather disastrous). And it especially annoying for automated CI pipelines.

This change updates our examples where requirements are added to also include apache-airflow==<version> in their requirements when they are adding them - this will keep airflow in the version that was there originally, and fail such installation with a conflict informatiion rather than downgrading/upgrading airflow silently.

Some of our users raised issues that when extending the image, airflow
suddenly started reporting problem with database versions and migration
not aplied or out-of-sync. This almost always turns out to be a
dependency conflict, that leads to automated downgrate or upgrade of
installed airflow version. This is - obviously - undesired (you should
be upgrading airflow consciously rather than accidentally). However
there is no way to do it implicitly - `pip` might decide to upgrade or
downgrade airflow as it sees fit. From the point of view - airflow is
just one of the packages and has no special meaning.

The only way to "keep" airflow version is to specify it together with
other requirements, pinned to the specific version. This PR updates
our examples to do this and explains why airflow is added there.

There is - of course - another risk that the user will forget to
update the version of airflow when they upgrade, however, sinc this
is explicit action performed during image extension, it is much easier
to diagnose and notice. We also warn the users that they should upgrade
when airflow is upgraded.
@potiuk potiuk force-pushed the add-instructions-to-keep-airflow-version-while-extending-image branch from c26eb83 to a2b36f2 Compare April 22, 2023 13:40
@potiuk potiuk merged commit bf6ebe9 into apache:main Apr 22, 2023
@potiuk potiuk deleted the add-instructions-to-keep-airflow-version-while-extending-image branch April 22, 2023 14:04
@ephraimbuddy ephraimbuddy added the type:bug-fix Changelog: Bug Fixes label Apr 23, 2023
@potiuk potiuk modified the milestones: Airflow 2.6.0, Airflow 2.6.2 May 21, 2023
@potiuk
Copy link
Member Author

potiuk commented May 21, 2023

cc: @ephraimbuddy -> while preparing my presentation about dependencies, I noticed this one has not been merged to 2.6.0 (Even though it has been marked as such), so I moved it to 2.6.2 (I have one more follow-up after that).

potiuk added a commit to potiuk/airflow that referenced this pull request May 21, 2023
It is important to keep airflow as a dependency with pinned version if
you are adding new dependencies. This PR adds it in case of
docker-compose documentation and also improves examples in our
images, by using AIRFLOW_VERSION variable defined in the images. This
way users can write future-proof ways of installind dependencies basing
on our examples.

Follow-up after apache#30813
potiuk added a commit that referenced this pull request May 21, 2023
…all (#31448)

It is important to keep airflow as a dependency with pinned version if
you are adding new dependencies. This PR adds it in case of
docker-compose documentation and also improves examples in our
images, by using AIRFLOW_VERSION variable defined in the images. This
way users can write future-proof ways of installind dependencies basing
on our examples.

Follow-up after #30813
potiuk added a commit that referenced this pull request May 23, 2023
…#30813)

Some of our users raised issues that when extending the image, airflow
suddenly started reporting problem with database versions and migration
not aplied or out-of-sync. This almost always turns out to be a
dependency conflict, that leads to automated downgrate or upgrade of
installed airflow version. This is - obviously - undesired (you should
be upgrading airflow consciously rather than accidentally). However
there is no way to do it implicitly - `pip` might decide to upgrade or
downgrade airflow as it sees fit. From the point of view - airflow is
just one of the packages and has no special meaning.

The only way to "keep" airflow version is to specify it together with
other requirements, pinned to the specific version. This PR updates
our examples to do this and explains why airflow is added there.

There is - of course - another risk that the user will forget to
update the version of airflow when they upgrade, however, sinc this
is explicit action performed during image extension, it is much easier
to diagnose and notice. We also warn the users that they should upgrade
when airflow is upgraded.

(cherry picked from commit bf6ebe9)
potiuk added a commit that referenced this pull request May 23, 2023
…all (#31448)

It is important to keep airflow as a dependency with pinned version if
you are adding new dependencies. This PR adds it in case of
docker-compose documentation and also improves examples in our
images, by using AIRFLOW_VERSION variable defined in the images. This
way users can write future-proof ways of installind dependencies basing
on our examples.

Follow-up after #30813

(cherry picked from commit d722fba)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:production-image Production image improvements and fixes kind:documentation type:bug-fix Changelog: Bug Fixes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants