Quartz64 | Migrate to kernel with more features

[MichaIng]

There are a couple of issues related to Peter Geis' kernel which has quite some limitations (missing kernel modules):

• Quartz64 | CONFIG_CAN_GS_USB is not set #6009
• Quartz64 | MicroK8s install fails #5962
• Quartz64 | br_netfilter module missing, breaking K3s #5886

It seems to be intended since he is not reacting to my merge requests to add those features.

I now tested a bunch of alternative kernel builds, but none of them is great:

• Armbian has the rk35xx kernel builds. These have device trees for all Quartz64 variants, but otherwise seem to be not meant for these SBCs, since major features, like onboard Ethernet, do not work.
• Armbian has the media kernel builds, used for the community-provided images as well. The current branch works actually okay. There is no dtb package (there are, but for older non-matching kernel versions only), but the device trees in /boot are embedded into the kernel package. It performs a little less good then Peter's in CPU and RAM read benchmarks, but that would be worth it. The problem is it will become edge.
• Edge suddenly requires an initramfs, since it does not have MMC drivers builtin. This doesn't make much sense since those SBCs cannot boot from anywhere else, AFAIK. So to not risk a sudden boot failure on kernel upgrade, we'd need to ship the images with initramfs. Not a big issue, but quite some disk space, writes etc, and we need another custom kernel postinst script to update a fixed symlink of the initramfs or extlinux. Also, it performs significantly worse on benchmarks, especially RAM reads, ~950 MiB/s vs 1400 MiB/s with Peter's kernel build. CPU is also about a second slower (11 vs 10). Not sure what the reason is. And on SOQuartz at least the CPU becomes very hot on idle, not sure why, probably too high default voltage. After a while of testing and reboots, it failed to boot until I let it cool down for a while.
• Armbian does not officially support all those. The community image builds are base on dedicated kernel and U-Boot builds based on RC versions. The repo doesn't ship any bootloader packages for model B and SOQuartz, and the model A U-Boot does not boot the others. So we'd need to stay with Peter's mainline U-Boot builds anyway.
• Generally it looks like a very unstable kernel basis, where sudden changes are likely (no provided community images use the kernel or bootloader builds from the repo, updates are not possible on them!), which do not follow the standards of other Armbian builds. There seem to be no progress towards official support, no one wants to be official Quartz64 maintainer, it seems.

I also tested the Debian kernel: It boots, but at least Linux 6.0 from Bullseye backports lacks major features. One cannot check or change CPU governor or see current frequencies, to start with. And it is ~400 MiB vs ~100 MiB, as a single kernel for ALL 64-bit ARM devices.

So what do I aim to do instead?

I'm gonna merge the needed changes for Peter's builds into my own fork, adjust the CI pipeline, hope that it works just the same way, so that we just do our own builds and updates. This is also an opportunity to gather some experience in kernel and bootloader builds.

[diederikdehaas]

If you want to try the Debian kernel (again), you need to have one from the 6.1 series in which several kernel modules for Quartz64 devices got enabled and later on several patches (mostly DT related) got backported from 6.2 to Debian's 6.1 kernel.

[MichaIng]

Okay, but this is applicable for Bookworm images only to get updates via APT. If we are lucky, the Bullseye backport kernel gets a bump to v6.1 as well.

[diederikdehaas]

Correct. I have no idea (if/)when a 6.1 backports will be made though. Best to periodically check https://tracker.debian.org/pkg/linux for that.

[diederikdehaas]

I just checked and all 3 mentioned issues should be resolved with Debian's 6.1 kernel:

$ grep -E "CONFIG_CAN_GS_USB|CONFIG_SQUASHFS_XZ|CONFIG_BRIDGE_NETFILTER" /boot/config-6.1.0-4-arm64 
CONFIG_BRIDGE_NETFILTER=m
CONFIG_CAN_GS_USB=m
CONFIG_SQUASHFS_XZ=y

[MichaIng]

Yes it does, also v6.0 and earlier versions. And all Armbian kernels have it as well. Peter's build is quite special in being very minimal 😉. Also the fact that he uses RC versions only shows that it is more for testing than for productive systems. I hope I'm able to rebase the kernel branch onto e.g. v6.1.8.

[MichaIng]

Own builds are running: https://gitlab.com/MichaIng/quartz64_ci/-/pipelines/776450060

Looks pretty good. Tomorrow I'll try merging Linux v6.1.11.

[diederikdehaas]

Shouldn't be too hard. Usually it's the upgrade to a new major version which can be difficult.

AFAICT Peter's primary 'concern' is getting things in the upstream kernel and for that you should base your patches on the most recent RC1 version.

[MichaIng]

Makes sense. Same for U-Boot. Another approach would be to just build with unmodified upstream Linux and U-Boot.

[diederikdehaas]

What do you mean by unmodified upstream (Linux)?
I don't have much experience with using upstream's defconfig's, but I'd assume they would be too limited for most. Certainly if you support multiple devices.

One major reason the Debian kernel for arm64 is (way?) bigger, is that it supports a whole bunch of different arm64 devices. Next to the 'standard features' which are present in all Debian's kernels.
And that all gets put into 1 kernel build.
If you only target (certain) Pine64 devices, which I guess Peter does, you can leave out all the other arm64 devices.

[MichaIng]

I mean the quartz64_defconfig from Peter with our additions. That one is copied in place by the CI pipeline and used for the build.

And another approach again would be (probably when Linux 6.2 has been released) to also use upstream defconf but applying customisation with a patch.

Ah I see now upstream is just a single arm64 defconf: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/configs/defconfig?h=linux-6.1.y

[diederikdehaas]

And another approach again would be (probably when Linux 6.2 has been released) to also use upstream defconf but applying customisation with a patch.

I'm not so sure you'd get what you want with that. It will add support for various devices which you don't seem to support.

[MichaIng]

Note the second part of my sentence "applying customisation" which implies removing support for all but the Quartz64 variants. We are still DietPi, so I'm gonna build an own kernel, it will have minimal unnecessary overhead 😉.

[MichaIng]

Good news:

• There seem to be no commits/changes in Peter's kernel branch which have not been merged upstream alreadyl. But I do not trust the GitLab GUI when starting a merge request for comparison, so doing this locally. But fetching upstream Linux takes ages 😄.
• Upstream U-Boot supports none of the Quartz64 models. I had a hard few hours to merge upstream v2023.01 and minimising the diff by removing parts related to Pinenote and Pinephone Pro, but now I have a branch which passes all tests: https://gitlab.com/MichaIng/u-boot

Bad news:

• I got mails from GitLab that my pipeline time is out. Sobering to know that they have quite a tiny limit. It's something between 5 - 10 hours per month, and afterwards you need to pay? I mean there is obviously not some huge company behind it like Microsoft, but then I wonder quite much why so many people praise GitLab for being the one and only and so much better than GitHub in all ways.
• Even stranger: The link to buy more pipeline time (I wanted to check prices) returns a 404, and I cannot find anything about it when browsing settings, so I'm very confused now.
• However, if I really cannot run the builds there, I just move them here.

[MichaIng]

There it is: https://about.gitlab.com/pricing/

400 minutes. That is done with 2-3 kernel builds 😄. Moving to GitHub tomorrow.

[MichaIng]

All done. U-Boot finally is only the stable v2022.04 (was rc1). I was able to migrate some parts for latest v2023.01, but others I wasn't able to figure out, especially the ATF (ARM Trusted Firmware) integration (actually optional(?) but nice to have). And also some SPL related RAM size/offset/address value which is suddenly needed and I don't know what to set it correctly.

However, images working now very well, only Quartz64 B test outstanding: https://dietpi.com/downloads/images/testing/

Kernel (and bootloader) can be upgraded with firmware packages from here: https://dietpi.com/downloads/binaries/
Will be offered as well next DietPi update.

[MichaIng]

The kernel configuration is now hosted here: https://github.com/MichaIng/DietPi/blob/dev/.build/images/Quartz64/quartz64_defconfig

This is kinda new land for me, so please give feedback about it. I'm sure there are still a lot of features missing, which one typically expects, respectively which is needed for any of all the software titles we offer. And on the other hand I'm sure there are features enabled which are not needed, or which are not needed builtin but can be made a kernel module, to keep the main kernel slim. I already found and disabled a bunch of features, like VirtIO drivers, legacy PTY nodes and such things, not sure how they made their way into this config in the first place.

[diederikdehaas]

Assuming the request was directed at me, I'm not sure what kind of feedback you're looking for.
And whether I can give feedback at all as my experience is with the Debian kernel and that uses a completely different build system.

What I do know is making your own initial kernel config is a huge undertaking. You need to figure out what you need and don't need/want and figure out how to make that work as 1 kernel module possibly depends on another, which depends on several others ... turtles all the way down.
I know for improving Debian's kernel config for Rock64, I made a LOT of kernel builds. And there all the groundwork was already done. I did look a lot at other projects kernel config, especially LibreELEC as I wanted to improve multimedia support.

[MichaIng]

Assuming the request was directed at me

Just to anyone who reads this. The new kernel solves a lot of reported issues/requests, so if anyone has some kernel config/build experience/knowledge, I'm happy for any input 🙂.

Debian kernel and that uses a completely different build system

It still uses a _defconfig in the end, isn't it? I can imagine that there are tools to manage it on an abstracted basis.

making your own initial kernel config is a huge undertaking

This is why I used Peter's which worked "well" (with the limitations) since we offer Quartz64 images, and extended/modified it as much as needed to solve reported requests and a little more about things I know about/looked up 🙂. No starting from zero intended.

what you need and don't need/want

Apart of all features we offer in our scripts and support for all software titles, that is indeed likely an iterative undertaking. This is why I am asking for input. There may be (there were, removed already) features enabled which are nonsense, probably only there for testing/debugging with downsides/overhead in production systems. VirtIO drivers are still some included, I didn't radically remove everything directly, but made them modules at least, the way they are on Debian. But I think they are actually pointless. There is CONFIG_ANDROID=y enabled, like on Debian, which is needed for Android emulations at least, which isn't needed, I'd say. But it has other impacts I read about, just not certain enough yet to remove it. And I have not much knowledge about all the encryption and hashing features, which ones are commonly used, which ones nearly never (can be made modules or removed) on modern Debian versions.

1 kernel module possibly depends on another

AFAIK this is internally solved. Linux has a dependency system, and I've seen features being available which are/were not explicitly enabled in your config, disabled by default, but found them to be needed for other features.

I know for improving Debian's kernel config for Rock64

Exactly what I am looking for 🙂. Not same SoC, but same manufacturer for SoC and SBC, so I lot of things apply to both the same way.

[diederikdehaas]

Debian kernel and that uses a completely different build system

It still uses a _defconfig in the end, isn't it? I can imagine that there
are tools to manage it on an abstracted basis.

No, upstream's _defconfig files are irrelevant for Debian.
Debian has 1 kernel for f.e. all arm64 devices and consists of:

• debian/config/config which has modules for all Debian kernels
• debian/config/kernelarch-arm/config which has (extra) modules for all arm
  based Debian kernels
• debian/config/arm64/config which has (extra) modules for arm64 devices

That's the general gist of things, but there are more factors which play a
role, like -rt or not. And possibly more; I still don't fully grasp it.

making your own initial kernel config is a huge undertaking

This is why I used Peter's which worked "well" (with the limitations) since
we offer Quartz64 images, ...

That helps ... for Quartz64 devices, but I thought DietPi supported more
devices? Or do they all get their own kernel (config)?
As said, it took me quite a while to properly enable a few options, while the
base was already fine. You need to make the config, build the kernel, verify
whether the kernel config matches what you intended and then test it to verify
it's working properly. And repeat that cycle for every feature you want.
Hopefully I'm wrong and it is simple though.

There is CONFIG_ANDROID=y enabled, like on Debian

FTR: it's no longer enabled in Debian's kernels.

1 kernel module possibly depends on another

AFAIK this is internally solved. Linux has a dependency system, and I've
seen features being available which are/were not explicitly enabled in your
config, disabled by default, but found them to be needed for other
features.

I don't know enough about the upstream build system, but I think it's
declarative and that the proper tooling, like menuconfig, can make use of
that. So with the proper tooling several things can be automated, but I'm not
so sure that Linux does that all automatically for you.

[MichaIng]

No, upstream's _defconfig files are irrelevant for Debian.

I do not mean upstream files, but that they use the same type of config files with CONFIG_ definitions inside. And they do. The files are not called _defconfig, but otherwise it's exactly the same: https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/config/arm64/config

That helps ... for Quartz64 devices, but I thought DietPi supported more devices?

The issue here is about the Quartz64 kernel only. In all other cases we do no own kernel builds but use Debian/Armbian/vendor kernel builds.

We may build more own kernels in the future, possible even taking the config from here as a reference, at least the learnings or find of features we definitely need (independently of hardware). But for now Quartz64 only.

You need to make the config, build the kernel, verify
whether the kernel config matches what you intended and then test it to verify
it's working properly. And repeat that cycle for every feature you want.
Hopefully I'm wrong and it is simple though.

Do not take me wrong, I'm happy for your input, but I know about general development procedures 😉. I really only need feedback about the individual config entries of the above linked file, hints about features we definitely do not need or those we do need, or do not need as builtins, as far as hardware-independent or known for Rockchip SoCs (like, well, they do not contain VirtIO devices). All onboard hardware features work fine already and all software titles which did not work before (without workarounds) do work now, so it's about fine tuning, disabling nonsense and enabling missing things we might not be aware of yet, e.g. possibly hardware acceleration for video features, encryption or hashing which is currently missing but just done without noise on software basis.

upstream build system

In the end, its make, and this seems to resolve config/module dependencies automatically. Also Debian's debhepler tools apply some patches and then run make: https://buildd.debian.org/status/fetch.php?pkg=linux-signed-arm64&arch=arm64&ver=6.1.12%2B1&stamp=1676661599

[MichaIng]

feedback about the individual config entries of the above linked file

EDIT: That one I mean: https://github.com/MichaIng/DietPi/blob/dev/.build/images/Quartz64/quartz64_defconfig

[MichaIng]

I'll close this issue now. Major issues have been solved and the new kernel + bootloader seem to work pretty well. Further optimisations can be tracked in dedicated issues.