Minimum supported Python 3 version bump

[jpakkane]

At some point we should update the minimum version of Python 3. Here's what the distros currently have:

• Debian stable, 3.7
• Ubuntu Xenial, 3.5
• Ubuntu Bionic, 3.6
• RHEL 8, 3.6???
• SLES 15, I could not find out by googling

Xenial is probably old enough that we don't have to keep supporting it. 3.6 seems to be widely available, but is that so. Are there platforms where we'd need to support 3.5?

[mensinda]

Practically speaking, what do we gain from upgrading to 3.6? Are there any special features we would need / like?

Otherwise, I don't see a specific reason to stop supporting Xenial...

[scivision]

tldr; my opinion is get type annotation checking throughout Meson before doing Python 3.6 refactoring. We may not be ready to gain significant benefits from making Python 3.6 a Meson minimum until late spring or summer 2020.

---

In the data science / aerospace / geoscience realm, I see robust adoption of Python 3.6. Generally my sense is people clung to 2.7 and then over the past 12 months have jumped straight to 3.6, due to issues with some big data science libraries not yet working with 3.7. Some significant libraries like xarray already have a Python 3.6 minimum.

However, on the hardware dev side of those industries, I see a slower adoption of new Linux distros. I see dev machines/servers hanging on to Ubuntu LTS (and Windows versions) to EOL and even a bit past EOL for offline systems due to driver/library compatibility issues and sluggish OEM SDK updates. They maybe could be brought to use non-system Python 3.6 in whatever form.

With Python 3.5 EOL in Sept 2020, we could take the next N months refactoring key pain points to stand ready for maximum internal Meson quality of life benefits from 3.6 like f-strings and widespread pathlib use. My first thought is we'd want to get type annotation checking more thoroughly implemented throughout Meson to avoid breakages in the refactoring.

Regardless of when the switch is, maybe we can think about what may need to change to get type annotation working almost everywhere in Meson for whenever 3.6 refactoring would occur -- the compiler mixins are one key area that's heavily interdependent.

[dcbaker]

Here's the list of new features in 3.6: https://docs.python.org/3/whatsnew/3.6.html

the big ones are inline variable annotations:

foo: typing.List[str] = []
vs
foo = []  # type: typing.List[str]

if also lets you do forward typing declarations:

foo: int
if os.system().startswith('linux'):
    foo = 2
else:
    foo = 3

An f-strings is the other big one.

f = 'foo'
'mystring : %s' % f
'mystring : {}'.format(f)
f'mystring: {f}'

Which is obviously really nice.

Speaking of compiler type annotations, I'm working on that right now.

[xclaesse]

Xenial is probably old enough that we don't have to keep supporting it

For the record, Ubuntu 16.04 (Xenial) support ends April 2021 and security support ends April 2024.
https://wiki.ubuntu.com/Releases

I personally don't think it should stop us from depending on Python3.6, but that's going to annoy some people for sure. I'm currently working for a client who still uses 16.04 and is not planning to upgrade any time soon. Again, I'm not saying this is a blocker, but we have to be prepared for complains, so better have solid reasons.

[xclaesse]

That being said, I can't wait to use f-strings !

[scivision]

I think that since Meson is by definition working heavily with strings, there is a real quality assurance benefit to end users from f-strings (avoiding internal mistakes).

Have we tried using PyInstaller for more than just .msi on Windows? That is, could we distribute using pyinstaller a standalone Meson "program" using PyInstaller for MacOS, Linux, BSD, etc. as well as the existing Windows Meson .msi. That would be for users without Python 3.6 already installed.

[jpakkane]

That's not the problem. People who use those old distros have to use the distro provided Python due to reasons. Otherwise they could just install their own Python version and use that.

[lazka]

There are pypi download stats btw, including Python versions: https://pypistats.org/packages/meson

[obilaniu]

@jpakkane If openSUSE Leap 15.1 is any guide to SLES 15 (and these days minor Leap versions are roughly twinned to SLES Service Packs), the current Python 3 in there is 3.6.

[scivision]

That plot naturally doesn't capture apt/yum, but then that category of use cases isn't as impacted by the switch--just whatever would hit their -backports.

It may be interesting to watch this graph for a while--will there be a bump from year-end/beginning system/python upgrades, and how rapidly will Debian Buster, Ubuntu 20.04, etc. be adopted into the summer. It could be nice to let all the refactoring settle out for a release cycle or two, before dropping Python 3.5, to leave a good quality last version for Python 3.5 users.

---

Maybe these internal quality improvements could warrant a Meson 1.0 release, dropping Python 3.5 or by Meson 1.1 or so dropping Python 3.5

---

Meson Python 3.4 -> 3.5 discussion

[amitdo]

See also #2891

[amitdo]

https://pypi.org/project/f2format/

[scivision]

From Twitter, recent benchmark showing f-strings can be a few times faster than .format()

#Python's f-strings are amazingly fast!

f'{s} {t}' # 78.2 ns
s + ' ' + t # 104 ns
' '.join((s, t)) # 135 ns
'%s %s' % (s, t) # 188 ns
'{} {}'.format(s, t) # 283 ns
Template('$s $t').substitute(s=s, t=t) # 898 ns

— Raymond Hettinger (@raymondh) December 14, 2019

A reply, noting how f-strings are "so fast"

1. It can calculate the length and the kind of the resulting string and allocate a memory only once.
2. It do not parse the format string at run time.
3. It do not create a tuple, all values are on the stack.
4. It do not look up an attribute and call a method.

— Serhiy Storchaka (@SerhiyStorchaka) December 15, 2019

[amitdo]

How about adopting a clear policy like this one:

https://numpy.org/neps/nep-0029-deprecation_policy.html

It does not have to be 42 months.

Even without such a policy, I suggest to inform users n months before dropping support for a specific python version.

[scivision]

Yes and NEP29 itself may need updating to account for new 12-month Python release cycle. NEP29 says to drop Python 3.6 support in July 2020--just a more aggressive cycle for data science.

A possible joint ruleset:

• don't support Python versions past EOL e.g. Python 3.5 EOL Sept 2020.
• support at least what the current CentOS release has in EPEL (Python 3.6)
• support the prior Ubuntu LTS Python version for at least one year past the current LTS release (e.g. support Python 3.5 from Ubuntu 16.04 at least until April 2019)

That's more complex than calendar months, but may be more representative of Meson's user base.

[amitdo]

support at least what the current CentOS release has in EPEL (Python 3.6)

You don't know when they will support a newer version of Python.

[scivision]

That's true, maybe calendar based is more suitable due to the sureness of date for that reason.

[dreamer-coding]

I found some information regarding python 3.6 on centos EPEL. Would this help?

https://cjsawer.whitewillow.co.uk/2018/08/29/epel-python-3-6-with-pip-on-centos-7/
https://www.2daygeek.com/install-python-3-on-centos-6/
https://www.rosehosting.com/blog/how-to-install-python-3-6-4-on-centos-7/

[scivision]

Another thought for a Python version deprecation ruleset:

• no later than when PSF deprecates (e.g. Sept 2020 for 3.5)
• when weekly downloads drop to less than say 10% for that old Python version for N out of M consecutive weeks

From the weekly downloads Python 3.5 was about 47% Nov 1 week, 35% Dec 1 week, and this last week about 26%. If that trend continued, maybe in February 2020 we'd have below 10% of Meson downloads being Python 3.5.

[scivision]

Looking at a few very popular projects, like Numpy and Requests, Python 3.5 is in the 15-20% range, with Python 3.6 the most popular. For Spyder IDE, Python 3.5 is in the 5-10% range with Python 3.7 by far the most popular. NEP29 deprecates Python 3.5 on Jan 7, 2020.

[xclaesse]

Given how fast multiple users complained when Meson 0.53.0 regressed on Ubuntu 16.04, I think that proof the point that we are not ready to bump dependency to >3.5.2, IMHO.

[scivision]

#6438 and associated discussion may provide a long-term solution for this. Having Meson be coded in a recent Python version syntax, while post-processing releases and perhaps even PRs/master to a rolling branch that's compatible with older Python versions.

[dreamer-coding]

Maybe this can help, link to a resource on downloading Python 3.6 on Ubuntu. This can be helpful if a Meson user would like to use a newer implementation of Meson that uses Python 3.6.

[gregsskyles]

This does seem crazy to me; why would people running an OS released nearly 4 years ago expect to install the latest release of a tool and feel like they shouldn't need to install a newer version of the tool's prerequisites? Especially since that's easy to do, as @michaelbadcrumble points out. If they must adhere to their release's outdated version of, in this case, python, then too bad, you can't have the latest version of your tool. Stay with the older version of meson that's compatible with your older environment.

[jpakkane]

As in most problems in the real world, the issue is not black and white, but instead there are shades of gray. There are many people doing development on LTS releases and every time we require something newer than is there by default, we are adding to their development burden. Thus the question is not about dropping all support or supporting everything perfectly forever, but instead working out a suitable point in time to do the bump so that we balance our desire to get Hot New Stuff vs not inconveniencing downstream developers too much.

[amitdo]

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/7.7_release_notes/index#enhancement_compiler-and-tools

New python3 packages are available in RHEL 7, which provide the Python 3.6 interpreter, as well as the pip and setuptools utilities. Previously, Python 3 versions were available only as a part of Red Hat Software Collections.

http://mirror.centos.org/centos/7/os/x86_64/Packages/

[amitdo]

https://www.phoronix.com/scan.php?page=news_item&px=RHEL-8.2-Beta-Released

RHEL 8.2 Beta Application Streams Bring GCC 9.1, Python 3.8

[gregsskyles]

Yes, fair enough. It is a bit of a high-wire balancing act, yes.

[amitdo]

SLES 15, I could not find out by googling

SLES 15 has python 3.6.

[jpakkane]

Bumping this again. Maybe the time has finally come? The only thing requiring 3.5 seems to be Xenial and since that is the old-old stable, I'd say we no longer need to support it.

[mensinda]

I would also like to add that there are still #6438 and #7518 around, both of which could be used to somewhat support 3.5.

[textshell]

repology seems to have versions for many distros: https://repology.org/project/python/versions

From the big ones: debian oldstable (stretch) (superseded since Juli 2019), raspian oldstable (superseded since Juli 2019), ubuntu < 18.04

[eli-schwartz]

If people still need meson on those distros, the best solution would be to make pyinstaller or similar, provide standalone binaries with embedded python. There are already similar resources for cmake and ninja.

[scivision]

For the last 6 months, PyPi downloads show ~10% of Meson downloads are for Python 3.5. As discussed earlier there are plenty of other important download pathways for Meson. Since I feel type annotation is important for any project and particularly for projects the size of Meson, and the type annotation is significantly better in Python 3.6 along with benefit of f-strings, this seems worthwhile.

Should we wait to take advantage of Python 3.6 syntax until there is next Meson release to ensure we can rollback if user backlash is too great?

[amitdo]

To upgrade the code to python 3.6, you can try to use these tools:

• https://github.com/ikamensh/flynt
• https://github.com/asottile/pyupgrade