aarch64 packages are only available for Erlang 26

[mnasiadka]

It's a bit of an issue for OpenStack Kolla - because we package older versions of RMQ than 3.12 - which require erlang 25.
I understand it's a limitation of the ppa on launchpad - but would it be possible to create a separate ppa for erlang 25?

[michaelklishin]

@mnasiadka the OpenStack community can set up any number of PPAs you need, the license of both Erlang and RabbitMQ, and these packages, allows for that.

[michaelklishin]

Launchpad supports just one (latest) release per distribution, by design.

ppa1.novemberain.com contains the entire archive of all releases we've done starting with one of the latest 24.3.x.

I don't understand why Team RabbitMQ is asked to set up another PPA on Launchpad while we already have solved many Erlang packaging woes. In fact, we've gone above and beyond what anyone else in the OSS community has done, and yet it is never good enough.

[acogoluegnes]

This is a follow-up to a conversation from another channel. The request here is to have arm64 packages available. They are available on PPA (built from source by the platform), but only for Erlang 26.0. They are not available on Cloudsmith because we build and upload those, but only for x86_64.

[mnasiadka]

Yes, basically as an OpenStack project we'd prefer not to build and package erlang (and maintain it) - and currently there is no repository available that has erlang 25.x for Debian Bullseye/Ubuntu Jammy arm64 (that is required for rabbitmq releases older than 3.12).

[michaelklishin]

So there are several (non-mutually exclusive) options:

1. Set up another PPA for Erlang 25 specifically (and around Erlang 27 release in 2024, another for Erlang 26)
2. Start producing Debian Erlang packages for aarch64, which means provisioning aarch64 Concourse workers
3. Try to build aarch64 packages in local Docker, like we do for our zero dependency RPMs

The second option can take a very long time.

For the last option out of these three, the problem is that there are many more packages, and more distributions, and the publishing process will be more involved. For the zero dependency RPM, it's a matter of uploading a few files to a GitHub release, thus covering nearly all RPM-based distributions under general maintenance. The picture is very different for Debian-based distributions.

Anyone who needs aarch64 packages can take this patched source package and follow any of the three options above. Option 1 seems to be the only realistic short term solution.

In general, option 1 seems to be the only viable short term solution.

[mnasiadka]

Looks like a good idea to go with 1, are you willing to do it and maintain it?
Sorry for the push, but currently we don't have a repository for arm64 erlang 25 - and that means no aarch64 builds for Debian/Ubuntu Kolla images (since RabbitMQ is a hard dependency on all of OpenStack).

[michaelklishin]

We already have a repo under the same account. Now we have to update our CI infrastructure to publish some packages there, and some to the original repo. That can take up to a week.

[mnasiadka]

Thanks! Can't wait ;-)

[michaelklishin]

It did not take as long as I expected it to #33 (comment).

[michaelklishin]

Never ceases to amaze me that when the OpenStack community needs something from us, they are very eager to talk.

When we solve their problem, they never report back or even say thank you.

[michaelklishin]

One semi-random idea I have is: we can mirror our Launchpad repos, too, and make ppa1.novemberain.com try all paths (so, all mirror copies) in order. There are two reason not to be particularly excited about this idea:

• apt-mirror might delete packages deleted from the upstream, I don't know for sure and we haven't deleted any packages from Cloudsmith ever since the mirror was introduced a bit less than a month ago
• An archive of aarch64 versions will be built slowly over time
• Finally, Debian distributions are not supported by PPAs, even though it's possible to use some Ubuntu PPAs on latest Debian

[noonedeadpunk]

I really do like this option though. As then you can use same source of packages and leverage apt pinning to weight sources or versions. On top of that it is slightly more in line with rpm, as using completely different approaches for rpm and deb is sometimes quite confusing, and syncing to novemberain makes it "universal" repo for both worlds.
And Debian still needs to be covered this way or another kind of, but would be good to cover at least Ubuntu first.
Also building archive over time is completely fair from my perspective. Way worse not having it at all.

[michaelklishin]

Glad that you like the idea but I'm done with Erlang packaging for the foreseeable future.

The source of this package is aarch64-ready, updated for Debian Bookworm, and generally ready to roll
for anyone and everyone who wants to host their own repository.

[michaelklishin]

Here are the new PPAs:

• for Erlang 25.3.x
• for Erlang 24.3.x

The original PPA will be used for the latest major and switched to Erlang 27 next May or so. When
that happens, a new PPA, rabbitmq-erlang-26 will be set up.

All PPAs have aarch64 package builds enabled.

It will take a few more hours for Ubuntu 22.04, 20.04, and maybe 18.04 packages to be published to these repositories. Otherwise they are ready to be tested.
repositories.

[michaelklishin]

I'm done working on Debian packaging of Erlang at least for a while. Debian does not have its own version of Launchpad, the fragmentation is worse than in the RPM world these days, the split of Erlang into 20+ packages is absolutely idiotic.

Equally importantly, the OpenStack community never pays a dime for all this work, and normally does
not even say thank you.

The source of the package is readily available, with a separate branch for every {version}-{distribution}. Building them in Docker is fairly trivial.

If you need aarch64 packages for Debian, please take the source and run with it. Anyone can set up a Cloudsmith or PackageCloud account, upload the packages built in Docker there using their CLI tools, and pay for the traffic themselves.

I'll go work on RabbitMQ instead.