-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add instructions on how to avoid accidental airflow upgrade/downgrade #30813
Add instructions on how to avoid accidental airflow upgrade/downgrade #30813
Conversation
This is a doc-only change - updating our examples of docs extending and customising. I think it might be worthwile to cherry-pick it to 2.6.0, as it should help our users to avoid surprising upgrades/downgrade of airflow when they are adding conflicting dependencies. This was a recurring pattern in quite a number of Slack threads and github issues, the problem is that This change updates our examples where requirements are added to also include |
Some of our users raised issues that when extending the image, airflow suddenly started reporting problem with database versions and migration not aplied or out-of-sync. This almost always turns out to be a dependency conflict, that leads to automated downgrate or upgrade of installed airflow version. This is - obviously - undesired (you should be upgrading airflow consciously rather than accidentally). However there is no way to do it implicitly - `pip` might decide to upgrade or downgrade airflow as it sees fit. From the point of view - airflow is just one of the packages and has no special meaning. The only way to "keep" airflow version is to specify it together with other requirements, pinned to the specific version. This PR updates our examples to do this and explains why airflow is added there. There is - of course - another risk that the user will forget to update the version of airflow when they upgrade, however, sinc this is explicit action performed during image extension, it is much easier to diagnose and notice. We also warn the users that they should upgrade when airflow is upgraded.
c26eb83
to
a2b36f2
Compare
cc: @ephraimbuddy -> while preparing my presentation about dependencies, I noticed this one has not been merged to 2.6.0 (Even though it has been marked as such), so I moved it to 2.6.2 (I have one more follow-up after that). |
It is important to keep airflow as a dependency with pinned version if you are adding new dependencies. This PR adds it in case of docker-compose documentation and also improves examples in our images, by using AIRFLOW_VERSION variable defined in the images. This way users can write future-proof ways of installind dependencies basing on our examples. Follow-up after apache#30813
…all (#31448) It is important to keep airflow as a dependency with pinned version if you are adding new dependencies. This PR adds it in case of docker-compose documentation and also improves examples in our images, by using AIRFLOW_VERSION variable defined in the images. This way users can write future-proof ways of installind dependencies basing on our examples. Follow-up after #30813
…#30813) Some of our users raised issues that when extending the image, airflow suddenly started reporting problem with database versions and migration not aplied or out-of-sync. This almost always turns out to be a dependency conflict, that leads to automated downgrate or upgrade of installed airflow version. This is - obviously - undesired (you should be upgrading airflow consciously rather than accidentally). However there is no way to do it implicitly - `pip` might decide to upgrade or downgrade airflow as it sees fit. From the point of view - airflow is just one of the packages and has no special meaning. The only way to "keep" airflow version is to specify it together with other requirements, pinned to the specific version. This PR updates our examples to do this and explains why airflow is added there. There is - of course - another risk that the user will forget to update the version of airflow when they upgrade, however, sinc this is explicit action performed during image extension, it is much easier to diagnose and notice. We also warn the users that they should upgrade when airflow is upgraded. (cherry picked from commit bf6ebe9)
…all (#31448) It is important to keep airflow as a dependency with pinned version if you are adding new dependencies. This PR adds it in case of docker-compose documentation and also improves examples in our images, by using AIRFLOW_VERSION variable defined in the images. This way users can write future-proof ways of installind dependencies basing on our examples. Follow-up after #30813 (cherry picked from commit d722fba)
Some of our users raised issues that when extending the image, airflow suddenly started reporting problem with database versions and migration not aplied or out-of-sync. This almost always turns out to be a dependency conflict, that leads to automated downgrate or upgrade of installed airflow version. This is - obviously - undesired (you should be upgrading airflow consciously rather than accidentally). However there is no way to do it implicitly -
pip
might decide to upgrade or downgrade airflow as it sees fit. From the point of view - airflow is just one of the packages and has no special meaning.The only way to "keep" airflow version is to specify it together with other requirements, pinned to the specific version. This PR updates our examples to do this and explains why airflow is added there.
There is - of course - another risk that the user will forget to update the version of airflow when they upgrade, however, sinc this is explicit action performed during image extension, it is much easier to diagnose and notice. We also warn the users that they should upgrade when airflow is upgraded.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.