Preparing the Raku Ecosystem for the Future

[lizmat]

This is an update of #39, with an account of the current situation and an explanation of all the pros and cons of each ecosystem backend.

Introduction

As of February 2022, the Raku Ecosystem Archive is fully operational and accessible with recent versions of zef after enabling it by installing Zef::Configuration and running zef-configure enable rea.

This means that modules can no longer "disappear" from the ecosystem, as a copy will be available in the REA. So it is no longer an issue should the p6c ecosystem or the CPAN ecosystem no longer be supported. You can in fact now already mimic that by running zef-configure disable p6c and zef-configure disable cpan. Authors would still be able to upload to these ecosystem, as the REA harvester will continue to scan these ecosystems to make sure any updates will become available in the ecosystem.

Overview

But module authors should be discouraged from using the p6c and CPAN for other reasons as well. So let's list the pros and cons of each ecosystem.

p6c

The original Raku ecosystem.

Pros

• it was very easy to implement
• an author does not need to signup for anything
• just need a URL for a META file that contains the download location of a distribution

Cons

• it is not clear who is responsible for the integrity of the module (unclear authority)
• can impersonate any author, there are no upload checks
• can have different versions with the same version value (thus breaking immutability principle)
• takes up to 4 hours to become visible to zef, which can be a pain in CI when working with multiple dependent distributions
• difficult to install any version other then the most recent (some 100+ modules in p6c do not have any version information at all!)
• modules can disappear without notice (unless already archived in the REA)

CPAN

Actually, not CPAN as most people know it, but using the file distribution network of CPAN to make Raku distributions available.

Pros

• integrated distribution upload support in e.g. App::Mi6
• the author is verified on upload of a distribution
• the version of a distribution is checked (immutability guaranteed)

Cons

• need to procure a PAUSE login if you don't have one already, which can be troublesome nowadays
• the authority of the module is not checked, so no guarantee that the uploader is responsible
• takes up to 4 hours to become visible to zef, which can be a pain in CI when working with multiple dependent distributions
• modules can disappear without notice (unless already archived in the REA)

zef (fez)

The name of the latest ecosystem is a bit confusing. The easiest way to think about it, is that it is called the "zef ecosystem", and that fez is the tool to upload (similar to PAUSE being the place to upload modules to CPAN).

Pros

• integrated distribution upload support in e.g. App::Mi6
• the author is verified on upload of a distribution
• the authority and version are checked on upload
• the distribution becomes available within seconds
• once uploaded, it can not be removed

Cons

• need to apply for a fez login (automated with email reply)
• it's a black box that lives in the cloud somewhere somehow

Recommendations

I'd like to therefore offer the following recommendations:

Disable p6c / cpan, enable rea by default in zef

This will make it clear that the p6c and cpan ecosystems are being phased out, while still not forcing module authors to immediately change their workflows with regards to module maintenance.

Publicize the sunsetting of the p6c / cpan ecosystems extensively

Now that we no longer need p6c and cpan to be available, we can announce that support for these ecosystems will be dropped on Jan 1st, 2023 (or another date not too distant in the future). It should be noted that this does not affect any modules currently in those ecosystems, as they will be available in the REA.

This would also need to include an extensive "how-to migrate to zef", more extensive than the faq documents.

Start creating issues in the remaining distributions

Any modules not having a more recent version in the zef ecosystem, should have issues created after July 1st, 2023 (6 months before the sunsetting date).

Sunset the p6c and cpan ecosystem updates

Stop the REA harvester to look at the p6c and cpan ecosystems. This would effectively ignore any updates done there. Module authors would be forced to migrate to the zef ecosystem. Should any of the unmigrated modules need a security / language version compatibility fix, then a higher numbered version of that module should be published in the Raku Language Module Adoption Center, which would effectively made that module Raku Community supported.

Conclusion

It should be clear that the p6c and CPAN ecosystems have had their use in the past, but that they are not ready for the future where stability and security become more and more important with applications based on Raku are being used in production.

[tbrowder]

Huzzah!

[jonathanstowe]

First up, in principle, I'm generally in favour of moving to a single better ecosystem backend, but:

Any modules not having a more recent version in the zef ecosystem, should have issues created after July 1st, 2023 (6 months before the sunsetting date).

Probably a typo but this is not consistent with:

support for these ecosystems will be dropped on Jan 1st, 2023

But assuming that the year is wrong in the former I'm not convinced that four months is sufficient before nagging people (at least myself.)

I have quite a few modules on CPAN, I'm planning on moving them all in one go when I'm ready to do so: I need to adjust my own tools and workflow to accommodate this.

I really would rather not get eighty or ninety issues on my modules however well meaning. If nothing else it's a waste of two people's time.

There are also a number of third party modules where I am the de-facto maintainer, these are going to have to be dealt with on a case by case basis.

So 👍 to the general principle of shifting the ecosystem but 👎 to encouraging putting issues on transgressing authors on an arbitrary deadline.

[raiph]

Any modules not having a more recent version in the zef ecosystem, should have issues created after July 1st, 2023 (6 months before the sunsetting date).

Typo?

CPAN PRO the author is verified on upload of a distribution

CPAN CON the authority of the module is not checked, so no guarantee that the uploader is responsible

Ooc, does author mean the uploader's login id (PAUSE id), authority mean the :auth<...> tag, and is the missing guarantee that the uploaded module's :auth<...> matches the uploader's login id?

zef (fez)

publishing-foo> fez foo = author's 💻 🢂 ☁️

installing-foo> zef foo = ☁️ 🢂 installer's 💻

[vrurg]

I really would rather not get eighty or ninety issues on my modules however well meaning. If nothing else it's a waste of two people's time.

Nothing actually urges a maintainer to migrate immediately. If a module is not getting its version updated it could remain as it is and be available via REA. But as soon as there is a need to release a new version, this could be used as an opportunity to migrate as well. A lazy sequence, you know... :)

[lizmat]

@raiph

Ooc, does author mean the uploader's login id (PAUSE id), authority mean the :auth<...> tag, and is the missing guarantee that the uploaded module's :auth<...> matches the uploader's login id

Indeed

@jonathanstowe

I really would rather not get eighty or ninety issues on my modules however well meaning. If nothing else it's a waste of two people's time.

Point taken. I guess it would be best if we could create one issue per module author. Also, please note that even after sunsetting p6c and cpan, your modules would continue to be available through the REA. It's just that when you need to update a module, it would have to move to the zef ecosystem in order to be "seen". So if modules are stable and not in need of an update, you wouldn't have to do anything.

[jonathanstowe]

I'm not proposing that I won't have them all migrated by January, just that I may not have any migrated by July 🤣 The overhead of doing them piecemeal doesn't warrant doing a new release to the new ecosystem in isolation if I haven't adapted my own tools, so they'll all be done at once.

[lizmat]

@jonathanstowe re: updating your tools. fez upload in the module directory, claims to be enough to upload to the zef ecosystem. Which would mean you'd only need to up the version and change the auth in the META6.json? Not trying to push you, you thought I'd mention it since we all can get set in our ways and not realize there are maybe other, possibly better ways to things nowadays. FWIW, I also like App::Mi6 a lot :-)

[cognominal]

I am not competent to give an opinion on the modes of module distribution…

No privilege or setup to access modules

… except that care must be made no privilege should be needed to access them. I thought I had filed an issue about that. But where ?

Here, not by me :
pierre-vigier/Perl6-Data-MessagePack#15

Time to learn how to use the vscode notebook extension to search issues across git repos.

https://marketplace.visualstudio.com/items?itemName=ms-vscode.vscode-github-issue-notebooks

User point of view

I propose to also see things from the user point of view.

in https://modules.raku.org/ there is no way to see which modules are part of rakudo star and no way to vote a module for inclusion. Or do I miss something ?

[CIAvash]

Probably out of scope of this issue, but I have a question regarding checking the integrity of uploaded/installed modules. Maybe @ugexe and @tony-o can answer that. Is there any package checking being done at the moment or is it possible? Like checking checksums/hashes. The goal of my question is can we verify that the package fez generates is the same as the package zef installs, in other words, can zef know the package hasn't been compromised on the server which hosts the package?

[melezhik]

Just my 2 cents, maybe off-topic, I could probably add a simple check to SparkyCI to verify that Meta6.json has "auth": "zef:author" ...

[patrickbkr]

@lizmat This issue is solved and can be closed, correct?

[lizmat]

Indeed :-)

[tony-o]

Probably out of scope of this issue, but I have a question regarding checking the integrity of uploaded/installed modules. Maybe @ugexe and @tony-o can answer that. Is there any package checking being done at the moment or is it possible? Like checking checksums/hashes. The goal of my question is can we verify that the package fez generates is the same as the package zef installs, in other words, can zef know the package hasn't been compromised on the server which hosts the package?

The name of the file is the md5 sum of the dist at the time of upload. As of now, there isn't an independent way of verifying the integrity unless it's built into the dist somehow