Python version constraints: common wisdom vs pragmatic realities

[aaronsteers]

History and background

It was not very long long ago at all that we did not have python version constraints that poetry provides. Version constraints for dependencies have existed for quite some time, but dependencies can be deployed anywhere, so long as they don't conflict with other constraints on the same package within the same virtual environment. (And even then, those sometimes those would be overridden or ignored depending on circumstance.)

Current situation

Unfortunately, the exact same "best practice" constraint policies of dependency management have been applied to environment runtime restrictions.

Now: If conventional best practices are followed, a Python package which has not been updated since the latest Python X.Y version was release cannot be run on Python X.Y unless and until the maintainer updates the repo to declare it to be so.

Worse: No Python packages or applications can be installed on newer Python versions until and unless the maintainers of all of their dependent packages have removed or bumped their Python version restrictions to include the newest Python release.

Worst case: If even a single important package is abandoned - even if the package is 'stable with no changes needed' - then all packages and applications which rely on that package will reach an artificial and premature 'dead end' of their own Python version support lifecycle.

Paradigm mismatch

The dependency constraint is designed to prefer and install only compatible versions of dependencies.

The runtime constraint (often conservatively and prematurely) prevents installation for the package in question and also all packages which might depend upon it.

Impact

The impact of conventional best practice of "always restrict to the unreleased next version" is that many, many, many Python applications and libraries will be dead-ended prematurely. This effect will increase over time as more and more packages and libraries adopt Poetry and strict Python version constraints.

Pragmatic Solution: change the best practice guidance to apply only 'major' Python version constraints for most use cases

For library maintainers:

1. Start treating 'minor' Python versions the same way as we treat 'minor' versions for other dependencies: ignore them unless and until they have known incompatibilities.
2. If no known incompatibilities exist for current/future python versions, then cap the Python version only at major Python versions.
3. As of today's date (Nov 2022), this would translate into a replacement of all <3.10 or <3.11 Python version constraints with a <4.0 constraint.
4. This will fail on libraries which depend upon libraries with more conservative "legacy" constraints.
   • Poetry should be amended with the ability to override those restrictions when a consuming library has tested and independently confirmed compatibility.
   • Until or unless the above is in place, Poetry will force library maintainers to apply the same minor version constraints as are applied to its most restrictive dependency.

For application maintainers:

1. If the application depends upon other Python applications, which are not fully testable from within CI, then continue with current best practices - which would recommend restricting the Python version to exclude whatever minor version is not-yet-released.
2. If the application is not heavily reliant on untestable dependencies, then apply the same revised guidance as above for libraries: i.e. replace future Python minor constraints with major constraints.

Revised Guidance in Practice

In practice, for the many types of Python packages we're concerned with for the Meltano community:

1. SDK: As a library, we should only apply major version constraints for future unreleased Python versions. (Aka, use <4.0 instead of <3.11).
2. Meltano itself: until and unless we can allow Meltano-managed Python apps to use an arbitrary python version when installed, then we should continue constraining to the next unreleased minor Python version. In other words, we prefer to not install Meltano at all if we are not confident that you'll be able to install and run Meltano's plugins and utilities.
3. Taps and Targets: If dependent libraries would allow it (and in most cases they won't, for the reasons explained above), taps, targets, and other utility 'apps' should start to constrain on future major Python versions instead of constraining on future minor versions.

Note / Caveat

While the above guidance seems at first bold (and perhaps controversial), the reality is that it may turn out to be mostly inconsequential, since as soon as we start to apply more relaxed constraints, many/most maintainers following this guidance will find that they cannot bump their max Python version constraint - that we are already blocked on bumping max Python version constraints until all of our dependencies are likewise bumped.

Hence, why this conversation is important.

[edgarrmondragon]

@aaronsteers You've probably read https://iscinumpy.dev/post/app-vs-library/ and https://iscinumpy.dev/post/bound-version-constraints.

I agree on all the points for the revised guidance 👍

[aaronsteers]

Yes. Both of these resources were influential - thanks for linking them here. 👍

And to be clear, as with everything, incremental changes followed by periods of testing/observation are probably a good way to go here. Behaviors in theory vs in practice sometimes give some surprise.

[WillDaSilva]

@aaronsteers I was unsure about your proposal at first (and to a small extent still am), but https://iscinumpy.dev/post/bound-version-constraints (thanks @edgarrmondragon!) makes a strong case for it.

I agree with the revised guidance.

[...] allow Meltano-managed Python apps to use an arbitrary python version when installed [...]

Do we have an issue open for this?

3. Taps and Targets: If dependent libraries would allow it (and in most cases they won't, for the reasons explained above), taps, targets, and other utility 'apps' should start to constrain on future major Python versions instead of constraining on future minor versions.

Maybe we can have Meltano try to force them to work with newer versions of Python by using pip install --ignore-requires-python.

[aaronsteers]

Maybe we can have Meltano try to force them to work with newer versions of Python by using pip install --ignore-requires-python.

Cool idea! I wonder if we'd perhaps pair this with meltano install --force since there are still some advantages of at least warning users if their tap doesn't officially support the version of Python we're running from.

[WillDaSilva]

I'll open an issue for this. EDIT: meltano/meltano#6975