The 2.0.0 Release

[felixfontein]

The 2.0.0 release will happen in the future. The main target(s):

• Remove PyOpenSSL backends (except for openssl_pkcs12) #273 Remove PyOpenSSL backends whereever possible
• Remove vendored copy of ipaddress #287 Remove vendored ipaddress in module_utils.
• Make Dirname (de)serialization conformant to RFC 4514 #274 Make dirName parsing / outputting more conformant to RFC4514
• Remove assertonly #289 Remove assertonly certificate backend
• Remove other deprecations #290 Remove other deprecated features scheduled for removal in 2.0.0

Timeframe: September/October 2021, 2.0.0 needs to be released by 2021-11-08 (https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/roadmap/COLLECTIONS_5.rst).

[MarkusTeufelberger]

Context on the timeline of the pyOpenSSL deprecation

This is mostly blocked by Ubuntu Xenial.

https://wiki.ubuntu.com/Releases - Xenial is EOS in April 2021 (and EOL 3 years later, but that's a bit far fetched since one needs to pay Canonical for support after EOS as far as I know).

pyOpenSSL (python-openssl) is version 0.15.1 there, with no newer version backported (https://packages.ubuntu.com/xenial/python-openssl)
cryptography (python-cryptography) however is only on version 1.2.3(!), also no backports available (https://packages.ubuntu.com/xenial/python-cryptography)

This means that we can't really drop support before that, since nearly nothing would work on such an old version of cryptography. All other distributions I looked at ship a more modern version of cryptography, so it is reasonable to make these modules depend on that instead. Unfortunately some functionality isn't (yet?) available there, so some modules might still require pyOpenSSL or rely on some vendored code, I guess we'll take a closer look in spring next year.

1.x support?

The nice thing about collections is of course that one can keep using the 1.x versions too, in case there's some older hosts around and as long as the changes are reasonable, I doubt that there's going to be much push-back on keeping a v1 branch around for a bit. We might need someone to actively maintain this though, if features should get backported etc., I personally don't want to do this and am looking forward to the day that I can rip out anything that mentions pyOpenSSL from the code base and not look back...

Other potential features for 2.0.0

I plan to get again into molecule to write a few tests for distribution compatibility of these modules until v2, now that the dust of their migration to a plugin based system maybe has settled a bit. This should also give a bit of signal to allow us to maybe remove some workarounds for versions of cryptography that are no longer around "in the wild".

https://github.com/ansible-collections/community.crypto/blob/main/plugins/module_utils/crypto/__init__.py states that it will go away eventually, 2.0.0 might be a good time for that.

Dropping support for Python 2 (and maybe requiring a certain patch version of Python 3 or higher) could also be possible, but I'd first like to see how ansible-base handles this on the controller side. While stuff like type hinting or f-strings would be great to have eventually, modules (especially the ones here that are used for quite basic tasks) need to be far more backwards-compatible than other parts of Ansible and I don't see huge wins to be had. We can surely discuss it, but my vote would be for a 3.0.0 release after 2.0.0 is out the gate with a discussion which minimum Python 3 version should be required (e.g. f-strings, type hints as comments or using more modern syntax)

[felixfontein]

Thanks for the update and your thoughts!

We need to still use pyOpenSSL for openssl_pkcs12 for some time I guess, since the new PKCS#12 support in cryptography is not good enough to replace pyOpenSSL yet, and it's so new that it will take a longer time until it's supported in the most common distros. But for all other modules, let's remove it for 2.0.0.

The "v1" branch will be called stable-1 (as in c.g and c.n repos), so we can (hopefully) use the backport bot to do most of the backporting. At least when there are no conflicts :) I think I'll try to mainly backport bugfixes, if applicable. If there are bugfixes that are hard to backport, I'd only work on it if they are important, or if someone requests them. If someone else wants to backport features and/or bugfixes, I won't stop them.

Removing https://github.com/ansible-collections/community.crypto/blob/main/plugins/module_utils/crypto/__init__.py in 2.0.0 (resp. removing its contents) sounds good.

About dropping support for Python 2: ansible-base will not drop support for Python 2 on the remote side for a long time, and I guess we shouldn't either (maybe not as long as they, but long enough). And since almost all of our code runs on that side, that means no new features for most of it :/ Eventually it should also be possible to declare programmatically that a collection does not support certain Python versions; right now it isn't. Anyway, 3.0.0 sounds good for that (if we have nothing else that's more urgent and requires us to do a 3.0.0 earlier).

[felixfontein]

How about scheduling 2.0.0 roughly next summer, so it can get included in Ansible 2.12? (More exact scheduling depending on when Ansible 2.12 will get scheduled :) )

[MarkusTeufelberger]

Yeah, according to https://docs.ansible.com/ansible/devel/roadmap/ROADMAP_2_11.html aiming for 2.11 will be just a little bit too early and I'm sure people will still keep Ubuntu 16.04 boxes around for a bit anyways. Aiming for summer 2021 (so there's enough time to test until everything lands in the 2.12 "full" release) is probably the best way forward.

[felixfontein]

BTW, that's the roadmap for ansible-base :) The roadmap for Ansible is at https://docs.ansible.com/ansible/devel/roadmap/COLLECTIONS_2_11.html It's pretty empty right now, but I know that the current planning is to get this out in the first months of 2021. (This also means that Ansible 2.11 will include ansible-base 2.10. Ansible 2.12 will include ansible-base 2.11.)

[felixfontein]

I think what we can also do is get rid of the vendored ipaddress.py file. ipaddress is part of the Python stdlib since Python 3.3, and can easily be installed for older Python versions (https://pypi.org/project/ipaddress/). It is also a requirement of newer cryptography versions (no idea right now how far back).

[felixfontein]

BTW, this is the current planning for the next Ansible releases: https://hackmd.io/y7BBcweNR3aRVLuMbKkDxw

3.0.0 will be released mid-February (feature freeze: February 2nd), and 4.0.0 (with ansible-core 2.11) is planned for mid-May (feature freeze: probably end of April for beta1). Ansible 5.0.0 will probably be released at the end of the year. ansible-core seems to be heading for a new release every ~6 months, and Ansible will probably follow a bit later (this is somewhat tentatively).

So I guess we should definitely have 2.0.0 out for Ansible 5.0.0, and for Ansible 4.0.0 it's probably a bit too early. So late summer/fall is probably a good time.

[felixfontein]

Getting rid of our vendored copy of ipaddress is definitely a good idea. Right now, our vendored copy protected us from CVE-2021-29921 (https://sick.codes/sick-2021-014), but I'd rather not have to maintain that code.

[felixfontein]

According to the Ansible 5 roadmap, community.crypto 2.0.0 needs to be released by October 25th (https://docs.ansible.com/ansible/devel/roadmap/COLLECTIONS_5.html) to get included in Ansible 5.0.0. I guess we can start somewhen in September by creating a stable-1 branch and start doing backwards incompatible changes in the main branch for 2.0.0.

[felixfontein]

According to the Ansible 5 roadmap, community.crypto 2.0.0 needs to be released by October 25th (https://docs.ansible.com/ansible/devel/roadmap/COLLECTIONS_5.html) to get included in Ansible 5.0.0. I guess we can start somewhen in September by creating a stable-1 branch and start doing backwards incompatible changes in the main branch for 2.0.0.

The schedule was postponed a bit, the deadline is now November 8th. I guess we can still start in September, but more at the end of it.

[felixfontein]

@MarkusTeufelberger @Ajpantuso the end of September is coming up :) Are there any more features that should go into 1.10.0, or should be make a 1.9.4 release for the current bugfixes in main and then start working on 2.0.0? (I'm not planning new features, so I'm happy with proceeding to start with 2.0.0 soon.)

[felixfontein]

Once we're done with 1.x.y releases, the procedure would be as follows:

1. Create a stable-1 branch from (then) current main
2. Merge Remove PyOpenSSL backends (except for openssl_pkcs12) #273
3. Work on other planned deprecations
4. Bump version in main branch to 2.0.0
5. Merge more PRs (like Make Dirname (de)serialization conformant to RFC 4514 #274); this can also be done earlier if they are ready (274 likely will)
6. Create a 2.0.0-a1 prerelease to allow easier testing
7. If we find issues, potentially have some more pre-releases
8. Once we're happy, release 2.0.0 (before November 8th)

Does that sound reasonable?

[Ajpantuso]

Are there any more features that should go into 1.10.0, or should be make a 1.9.4 release for the current bugfixes in main and then start working on 2.0.0?

I'm okay to proceed without anything additional in 1.10.0.

[MarkusTeufelberger]

Same here, looking forward to (mostly) ripping out pyOpenSSL! :-)

[felixfontein]

Great! I'll do the 1.10.0 1.9.4 release today then, and then create stable-1 and merge #273 and #274 (assuming the tests still pass). Then we can proceed with other cleanup :)

@MarkusTeufelberger I hope you haven't been waiting to rip parts out yourself, since I'm afraid I already killed most of it that can be removed for 2.0.0 in #273 ;)

[felixfontein]

Ok, 1.9.4 is out, stable-1 exists (its regular CI run happens only once per week, as opposed to daily for main), and #273 and #274 have been merged 🎉

[felixfontein]

I'll create a 1.9.5 release tomorrow from the stable-1 branch so we have a release that supports cryptography 35.0.0.

[felixfontein]

1.9.5 is out.

[felixfontein]

The only larger piece required for 2.0.0 is #290. Once that's merged, the main branch should mostly be ready for 2.0.0. Something we should decide on before making a first (pre-)release is whether we want to address #76 by adding a new module or not. If we want to do that, we can re-use the removed code (from #289) and adjust #289's removed feature changelog fragment by mentioning the new name. Now that it's 1.5 years ago that we last discussed #76, I think it's time to come to a conclusion on this question.

[felixfontein]

Ok, I guess we can proceed with 2.0.0 soon. Does anyone mind if I create a 2.0.0 release this weekend? Or does someone wants to wait a few days more (say, the weekend thereafter)?

[felixfontein]

If nobody minds, I will begin with the 2.0.0 release once #316 and #318 are merged. Would be great if someone could review them!

[felixfontein]

2.0.0 has been released! 🎉