-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enable multi-arch pipeline #262
Comments
Fedora has multi-arch hardware, although I think most of it is "owned" by Koji. We'd need to either liberate some or make some machines "dual role" as a Jenkins worker too. For aarch64 there's packet.net at least. |
See this cosa issue for some previous work (and notes for further work): coreos/coreos-assembler#673 |
@bgilbert is there a project for things to do after stable? I don't think this should block stable but we need to track it |
@dustymabe There isn't. Do we have a specific timeline in mind? If not, there's always the fact that it's in this tracker. 🙂 |
I was mostly looking for a way to categorize things as "important, but after stable" that could separate it from the rest of the open issues. |
OK, so there are three rough suggestions so far on ways to farm out work to arch-specific machines:
Of those, I'd much prefer the first option. At least I hope we aim for the first option first before falling back on the other ones. I really don't want to go back to maintaining Jenkins workers. |
I'd definitely agree on preferring the first option. |
One note; since today source-of-truth is S3, and S3 doesn't have a defined way to do locking or even compare-and-exchange we need application-level handling for this. In other words, one Jenkins instance "owns" |
That doesn't really exist as far as I know. I don't think we can depend on making that work. Maybe something like Federation would allow us to continue using the Jenkins Kubernetes plugin? You're missing another whole class of option, which is to have a separate pipeline per architecture, but have the alt-arch pipelines submit "merge requests" to the main x86_64 one. A bit like koji-shadow perhaps. |
Yeah, I think there's an obvious bootstrapping issue of sorts I ignored: trying to run on an OpenShift multi-arch cluster to hack on software which will enable OpenShift multi-arch clusters... It's not clear to me how well supported OpenShift v3 is on non-x86_64 nodes (looks like ppc64le is at least, it was hard to find information about aarch64 and s390x). So I guess to start with we'll have to hack around this (but definitely revisit once OCP/Origin v4 is in a better position).
Yeah, it does seem to me like the simplest solution is to just reuse the multi-arch hardware from Koji/Brew. I filed https://pagure.io/fedora-infrastructure/issue/8370 to start a discussion around this. Feel free to add ideas there! |
For ppc64le it would be the same as x86_64, for s390x/aarch64 we do build the containers and you can install it, but it does take some additional settings (e.g. pointing to the location of the container images and some additional repositories before running the openshift-ansible setup). |
One thing related to this is AIUI the current plan for Fedora IoT is to continue using Pungi, maybe feeding the ostree commit into osbuild. |
A workaround here would be to run a full aarch64 cosa container on x86_64 using qemu-user-static. This would be slower from a performance perspective but not as slow as using a full system emulation. |
A few of us met recently to discuss multi-arch + FCOS. Here is a summary of that meeting:
|
First steps should be tracked by #541 |
+1. interested to follow and help especially for s390x/ppc64le. |
Per https://github.com/projectatomic/rpmdistro-gitoverlay/blob/master/doc/reworking-fedora-releng.md#blend-upstream-testing-and-downstream-testing I think Koji should be reworked to run on Kubernetes/OpenShift, then we only have one cluster. |
IIUC a Koji Operator is part of what's being worked on in https://github.com/fedora-infra/mbbox (thanks for the pointer to the repo @Conan-Kudo!) |
That looks like it's just talking to Koji to schedule builds. I am more talking about doing the same architecture we have with the FCOS pipeline where our workers are just pods. This would really go towards dropping mock for example. |
This all said, it probably wouldn't be really hard for us to support in the pipeline a model where we just get SSH access to a system and we run cosa via podman directly. That'd let us pretty easily spin up e.g. an aarch64 machine in AWS to run a build, wouldn't need Kubernetes or even FCOS. |
👍 |
@dustymabe The work I've been doing should help here a lot. Not sure what the interest is for multiarch, but the multiarch story is the one that Gangplank (formerly Entrypoint) specifically seeks to address. |
Poseidon is publishing some arm64 AMIs to a few regions for Typhoon Kubernetes experimental ARM64 support until there are official images, which I'd be excited to see! Hopefully greater access will spawn more interest and testing. But the images may help in a pinch for those who can't build their own and need to bridge the gap (usual caveats about not trusting AMIs from strangers). |
I am trying to build fedora CoreOs for aarch64, Following is the configuration of my machine:
I am following steps from the link. The cosa init step is running successfully but for cosa fetch step I am getting error as "fatal: Missing /dev/kvm". The output of kvm-ok is as following: Can you please provide me any pointers on resolving this issue? Can we skip using virtualization device and build image? Thanks. |
Are you sure |
Mostly likely due to insufficient privileges, try chmod 777 on /dev/kvm (for a quick test only). If you do want to disable kvm you can add environment variable COSA_NO_KVM=1 to the podman command, however the performance will be unacceptable and even hang forever on specific platforms. |
@odidev I would look in the direction of what @dustymabe and @NickCao mentioned. I can say that I have no issue building the FCOS via cosa(in the same way as mentioned in the link) on the old x-gene, rk3399(rockpro64, small cores need to be off-lined for KVM/qemu to work) or rpi4 all running various Fedora versions(32-rawhide). |
@NickCao thanks for your suggestion, after giving sufficient privilege, I was not facing the issue with the KVM. @jcajka I built coreos-assembler image using Dockerfile. I was able to run all the steps mentioned in the link. And the container was running successfully. Following is the output of "uname -m": |
I am sure I mentioned this before but am not seeing it now. AIUI Fedora also runs an OSBS instance; conceptually our workload could run in the same OpenShift instance I believe. |
are there aarch64 builds today? if so, can someone steer me to them? |
https://fedorapeople.org/groups/fcos-images/builds/
But most likely not what you are looking for?
…On Wed, Mar 10, 2021 at 11:48 AM Brent Baude ***@***.***> wrote:
are there aarch64 builds today? if so, can someone steer me to them?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#262 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAKDYHRUZJMK7E4NWJLNMKTTC65JRANCNFSM4IPSVL4A>
.
|
Yes, there are. They're used for flatpak/container pipeline builds. |
Update. We have an aarch64 node that we then use gangplank to connect to and execute builds (still working with Fedora infra to possibly get an AARCH64 node within that DC). We are now bumping the lockfiles for aarch64 as a result of the following PRs:
Next step is to enable an aarch64 pipeline that's triggered as part of the normal pipeline runs. |
Since we are shipping aarch64 artifacts now we can mark this one as done. |
See this comment:
Credit to @arithx for the suggestion.
The text was updated successfully, but these errors were encountered: