6.2.8 kernel revert needed for zfs-2.1.9 dkms #14658

comrade-meowski · 2023-03-23T02:43:30Z

The latest stable kernel 6.2.8 causes zfs-2.1.9 & zfs-2.1.10-staging to fail during dkms installation of the zfs modules.

The error given in the dkms make.log is:

ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_start_io_acct'
ERROR: modpost: GPL-incompatible module zfs.ko uses GPL-only symbol 'bio_end_io_acct_remapped'

Building 6.2.8 with commit b477180587d81a4dd756361098bd534dfa6bb53d reverted allows zfs dkms to complete normally again. I have only tested with dkms but presumably this will more generally stop zfs building against 6.2.8 at all.

This is the kernel commit:

commit b477180587d81a4dd756361098bd534dfa6bb53d
Author: Yu Kuai <[email protected]>
Date:   Thu Feb 23 17:12:26 2023 +0800

    block: count 'ios' and 'sectors' when io is done for bio-based device
    
    [ Upstream commit 5f27571382ca42daa3e3d40d1b252bf18c2b61d2 ]
    
    While using iostat for raid, I observed very strange 'await'
    occasionally, and turns out it's due to that 'ios' and 'sectors' is
    counted in bdev_start_io_acct(), while 'nsecs' is counted in
    bdev_end_io_acct(). I'm not sure why they are ccounted like that
    but I think this behaviour is obviously wrong because user will get
    wrong disk stats.
    
    Fix the problem by counting 'ios' and 'sectors' when io is done, like
    what rq-based device does.

The text was updated successfully, but these errors were encountered:

philmmanjaro · 2023-03-23T03:09:25Z

That patch got originally added to 6.3-rc3 kernel and backported to 6.2.8

rincebrain · 2023-03-23T07:38:46Z

That's probably fixable, I would expect, based on reading the commit - I would guess that the difference is the signature change is making some kernel compat configure check fall back on something that's a GPL-only export now, and a simple update to that would resolve this...

e: something like 3ace9de (untested on 6.2+, I don't have that tree laying around conveniently)

e2: or more like the above plus 2e9c9b7

simonbcn · 2023-03-23T07:57:13Z

Arch Linux
Linux-xanmod 6.2.8

Same problem compiling from source.

wtosta · 2023-03-23T08:26:39Z

I confirm the issue on SlackWare Linux 15 x86_64 current compiling ZFS on 6.2.8 kernel. On Linux kernel 6.2.6 I haven't got any compilation issues.

ptr1337 · 2023-03-23T14:17:08Z

Same issue here.
Log is in this action:
https://github.com/CachyOS/linux-cachyos/actions/runs/4492407990

HougeLangley · 2023-03-23T14:42:12Z

Same issue

https://bugs.gentoo.org/902821

demmm · 2023-03-23T15:13:56Z

That's probably fixable, I would expect, based on reading the commit - I would guess that the difference is the signature change is making some kernel compat configure check fall back on something that's a GPL-only export now, and a simple update to that would resolve this...

e: something like 3ace9de (untested on 6.2+, I don't have that tree laying around conveniently)

e2: or more like the above plus 2e9c9b7

Both patches apply cleanly on 6.2.8, but no change in the build error, seems versions not used when applied to 6.2?
See messages:

checking whether blk_cleanup_disk() exists... no
checking whether generic bdev_*_io_acct() are available... no
checking whether generic disk_*_io_acct() are available... no
checking whether generic bio_*_io_acct() are available... yes

according to the patch, second line should be whether 6.2+ bdev_*_io_acct() are available

rincebrain · 2023-03-23T17:02:10Z

Did you regenerate with autogen.sh after applying before rerunning configure? Because at least for me, on the checkout of 2e9c9b7, on a 5.15 system: configure:110409: checking whether 6.2+ bdev_*_io_acct() are available configure:110424: result: no configure:110426: checking whether pre-6.3 bdev_*_io_acct() are available configure:110441: result: no configure:110443: checking whether generic disk_*_io_acct() are available configure:110450: result: yes

…

On Thu, Mar 23, 2023 at 11:14 AM demmm ***@***.***> wrote: That's probably fixable, I would expect, based on reading the commit - I would *guess* that the difference is the signature change is making some kernel compat configure check fall back on something that's a GPL-only export now, and a simple update to that would resolve this... e: something like 3ace9de <3ace9de> (untested on 6.2+, I don't have that tree laying around conveniently) e2: or more like the above plus 2e9c9b7 <2e9c9b7> Both patches apply cleanly on 6.2.8, but no change in the build error, seems versions not used when applied to 6.2? See messages: checking whether blk_cleanup_disk() exists... no checking whether generic bdev_*_io_acct() are available... no checking whether generic disk_*_io_acct() are available... no checking whether generic bio_*_io_acct() are available... yes according to the patch, second line should be whether 6.2+ bdev_*_io_acct() are available — Reply to this email directly, view it on GitHub <#14658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7ME5NXK7VDY2XWLGETW5RSD7ANCNFSM6AAAAAAWETO4DM> . You are receiving this because you commented.Message ID: ***@***.***>

demmm · 2023-03-23T17:05:55Z

@rincebrain That was it (I thought it was enabled, but ended up being commented out), builds fine now.

Linux 6.3+, and backports from it, changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Fixes: openzfs#14658 Signed-off-by: Rich Ercolani <[email protected]>

jonhermansen · 2023-03-24T20:34:11Z

Is it fairly common that Linux removes or prevents access to certain symbols that zfs relies on? I recall this happening in the past, with encryption primitives...

Linux 6.3+, and backports from it, changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Fixes: openzfs#14658 Signed-off-by: Rich Ercolani <[email protected]>

behlendorf · 2023-03-24T21:51:33Z

The kernel ABI is always a work in progress and it evolves as needed by the upstream kernel developers. This means for each new kernel release the OpenZFS code may need to be updated to accommodate recent kernel changes. It's for this reason OpenZFS includes an extensive set of configure checks to determine what interfaces are available.

Generally speaking it's fairly uncommon for this kind of breaking change to be backported to a released kernel. When it does happen the fix usually involves updating our checks to detect the modified interface. That's in fact what happened in this case. It's much less common for an interface to be unexpectedly removed or converted in some way which makes it unavailable. However, it does happen on occasion and it's something to keep in mind if you're running a more bleeding edge kernel. This is far less likely to occur with any of the enterprise distributions (Ubuntu LTS, RHEL, SUSE, etc).

We'll get this fixed up in the 2.1.10 release.

rincebrain · 2023-03-24T22:50:21Z

Also, as a footnote, I don't believe it's ever been the case that OpenZFS consumed the Linux kernel's crypto interfaces - you may be thinking of the FPU interfaces.

…

On Fri, Mar 24, 2023 at 5:51 PM Brian Behlendorf ***@***.***> wrote: The kernel ABI is always a work in progress and it evolves as needed by the upstream kernel developers. This means for each new kernel release the OpenZFS code may need to be updated to accommodate recent kernel changes. It's for this reason OpenZFS includes an extensive set of configure checks to determine what interfaces are available. Generally speaking it's fairly uncommon for this kind of breaking change to be backported to a released kernel. When it does happen the fix usually involves updating our checks to detect the modified interface. That's in fact what happened in this case. It's much less common for an interface to be unexpectedly removed or converted in some way which makes it unavailable. However, it does happen on occasion and it's something to keep in mind if you're running a more bleeding edge kernel. This is far less likely to occur with any of the enterprise distributions (Ubuntu LTS, RHEL, SUSE, etc). We'll get this fixed up in the 2.1.10 release. — Reply to this email directly, view it on GitHub <#14658 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7JEQHUZA76HXQ73KMLW5YJPBANCNFSM6AAAAAAWETO4DM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

rincebrain · 2023-03-24T22:57:46Z

I just cut #14668, but still don't have a 6.3 tree handy, could someone check that I didn't break it in cleaning up various style nits?

kode54 · 2023-03-26T08:43:49Z

Links against 6.2.8, but I want to make sure I build a fresh 6.3 before I test that. The last one I successfully built was rc1, and I don't know if they've changed anything significantly since then. That shouldn't take too long, though.

ptr1337 · 2023-03-26T08:50:20Z

Links against 6.2.8, but I want to make sure I build a fresh 6.3 before I test that. The last one I successfully built was rc1, and I don't know if they've changed anything significantly since then. That shouldn't take too long, though.

I've tested to compile the above commit against 6.3rc3 and it did not work at all.
inode_owner_or_capable() is failing.

Above commit against 6.2.8 worked fine.

imrehg · 2023-03-26T13:17:47Z

Pretty related to #14555.

rincebrain · 2023-03-26T13:38:58Z

Not really, no. That's an AArch64-specific breakage that's unrelated except also in a new kernel.

lunasophia · 2023-03-26T21:50:50Z

Is there anything I can do to help get this applied and released? I can't boot my machine because of this issue.

Alanaktion · 2023-03-26T22:43:05Z

Is there anything I can do to help get this applied and released? I can't boot my machine because of this issue.

Switching to the linux-lts kernel package with zfs-dkms builds fine for now and allows booting normally, as the backported change only affects more recent kernels.

kode54 · 2023-03-26T23:13:39Z

Is there anything I can do to help get this applied and released? I can't boot my machine because of this issue.

Switching to the linux-lts kernel package with zfs-dkms builds fine for now and allows booting normally, as the backported change only affects more recent kernels.

Not terribly useful to use LTS if you have bleeding edge hardware like I do.

lunasophia · 2023-03-26T23:21:30Z

Also not useful if you've upgraded your pools to have features that aren't in the modules in the current zfs-dkms version. This is why I offered to help more directly instead of asking for a workaround. I'm on Arch and have been tracking zfs-dkms-git so I can have an up to date kernel. I can't use an older version, I need a release to be able to boot my machine. Again, can I help accelerate this at all?

rincebrain · 2023-03-26T23:27:35Z

Apply #14668 locally if you need this fixed more urgently than releases come out?

If you want to run the latest kernel, there's always going to be a timelag between Linux merging breaking changes and them being fixed, in git or in a release branch, in OpenZFS, which is why I usually advise people don't run on a continuously-updating latest Linux kernel versus a known quantity if they need to rely on OpenZFS.

isopix · 2023-03-26T23:29:29Z

Also not useful if you've upgraded your pools to have features that aren't in the modules in the current zfs-dkms version. This is why I offered to help more directly instead of asking for a workaround. I'm on Arch and have been tracking zfs-dkms-git so I can have an up to date kernel. I can't use an older version, I need a release to be able to boot my machine. Again, can I help accelerate this at all?

Here it goes:

diff --git a/PKGBUILD b/PKGBUILD
index 6e04436..badc10c 100644
--- a/PKGBUILD
+++ b/PKGBUILD
@@ -10,7 +10,7 @@
 #
 
 pkgname=zfs-dkms-git
-pkgver=2.1.99.r1726.g7883ea2234
+pkgver=2.1.99.r1854.ga05263b7aa
 pkgrel=1
 epoch=2
 pkgdesc='Kernel modules for the Zettabyte File System.'
@@ -19,15 +19,18 @@ url='https://zfsonlinux.org/'
 license=('CDDL')
 groups=('zfs-git')
 makedepends=('git')
-provides=("ZFS-MODULE=${pkgver}" "SPL-MODULE=${pkgver}" "${pkgname%-git}=${pkgver}" 'spl-dkms')
+provides=("ZFS-MODULE=${pkgver}" "SPL-MODULE=${pkgver}" "${pkgname%-git}=${pkgver}" 'spl-dkms' "zfs" "zfs-headers" "spl" "spl-headers")
 conflicts=("${pkgname%-git}" 'spl-dkms')
 replaces=('spl-dkms-git')
 source=("git+https://github.com/openzfs/zfs.git"
-        "0001-only-build-the-module-in-dkms.conf.patch")
+        "0001-only-build-the-module-in-dkms.conf.patch"
+	"zfs_kernel6.2.8+.patch")
 sha256sums=('SKIP'
-            '539f325e56443554f9b87baff33948b91a280ec1daadcb0c636b105252fcd0f5')
+            '539f325e56443554f9b87baff33948b91a280ec1daadcb0c636b105252fcd0f5'
+            'ce62c656444a0456291f6491795cd7629b8f7621a08f8c3bc51dd77788a40a5e')
 b2sums=('SKIP'
-        'a8ab5da81d214e7801f0f8cdf77c076c714a3f17292df15ca35fcf7aef2c4d505348797e3b1da7590ea303ff488490ddba49e6f9e3f8a0bcc975894d51d97c2b')
+        'a8ab5da81d214e7801f0f8cdf77c076c714a3f17292df15ca35fcf7aef2c4d505348797e3b1da7590ea303ff488490ddba49e6f9e3f8a0bcc975894d51d97c2b'
+        '0a91acccebbc33631343b627c8d0c5ce339f4a725682c62323e190ad560f4ffaeeebcaa79e93754dbd325133994922d1aab5add70a64bf0bc1ac1b6cf48d7b51')
 
 pkgver() {
     cd zfs
@@ -39,6 +42,7 @@ prepare() {
     cd zfs
 
     patch -p1 -i ../0001-only-build-the-module-in-dkms.conf.patch
+    patch -p1 -i ../zfs_kernel6.2.8+.patch
 
     # remove unneeded sections from module build
     sed -ri "/AC_CONFIG_FILES/,/]\)/{

for "zfs_kernel6.2.8+.patch" it's there:
https://903117.bugs.gentoo.org/attachment.cgi?id=859043
from there:
https://bugs.gentoo.org/903117

admnd · 2023-03-26T23:33:22Z

Apply #14668 locally if you need this fixed more urgently than releases come out?

Applied onto ZFS 2.1.9 and works wonderfully well here even with Linux 6.2.8. Good job @rincebrain.

The aforementioned Gentoo bug #903117 is nothing more than changes brought in by PR #14668 put together in a single patch file.

isopix · 2023-03-26T23:49:54Z

Apply #14668 locally if you need this fixed more urgently than releases come out?

Applied onto ZFS 2.1.9 and works wonderfully well here even with Linux 6.2.8. Good job @rincebrain.

The aforementioned Gentoo bug #903117 is nothing more than changes brought in by PR #14668 put together in a single patch file.
you are right. I was thinking it's archzfs github, but its openzfs ;-)

bsdice · 2023-03-27T00:04:48Z

Fellow Arch user here, I also got bit by this. Currently using zfs-dkms on 6.2.8 with reverted b477180587d81a4dd756361098bd534dfa6bb53d. My thanks to @comrade-meowski for identifying the offending commit.

I also use linux and not linux-lts because imho "LTS" is not really stable. Well, "stable" is not stable we just found out. ;-) There's a constant stream of backports introducing regressions, which then nobody figures out. In the tree close to what Torvalds releases, I think more testing is done. Since I need a bunch more kernel patches (debug symbols, wifi, mellanox, etc.) I use a script to build patched kernels through auto-patched PKGBUILDs anyway. Learned this trick to create a reversal patch from a forward patch:

interdiff -q 628-zfs.patch /dev/null > 628-rev-zfs.patch

To add, thanks for the advice to not discuss on LKML or elsewhere. Obliging... Just wish a big distro like Arch would upgrade ZFS to a first class citizen, possibly endorsing zfsbootmenu to boot zfs snapshots straight from a menu out of UEFI. Once people are hooked on snapshots, encryption, compression, CoW, checksums for everything, I'd hope upstream would no longer be able to ignore the only filesystem I trust.

kode54 · 2023-03-27T02:39:22Z

With the above linked PR, you don't need to modify the kernel in any way to use 6.2.8.

Also, #14622 is relevant to further impending ~~doom~~ fun for anyone who wants to get involved, beyond the kind person in the issue who reported to be working on it.

youzhongyang · 2023-03-27T03:09:31Z

Yes, a PR for #14622 will soon be ready for review. Test suite run has passed on 6.3-rc2, now running on a 5.15 kernel.

Linux 6.3+, and backports from it (6.2.8+), changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rich Ercolani <[email protected]> Closes openzfs#14658 Closes openzfs#14668

Linux 6.3+, and backports from it (6.2.8+), changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rich Ercolani <[email protected]> Closes #14658 Closes #14668

andr31g · 2023-03-30T04:24:49Z

For those who just want a quick rundown of the steps involved, here is what I did (Fedora 37 Workstation):

(Note that the system was previously building the zfs dkms package just fine, and so has all the necessary pre-requisites already installed, such as e.g., kernel-devel, etc)

Thank you for the quick fix @rincebrain!

# dnf install autoconf automake libtool
# cd /usr/src
# wget https://patch-diff.githubusercontent.com/raw/openzfs/zfs/pull/14668.patch
# cd zfs-2.1.9/
# patch -p1 -i ../14668.patch
# ./autogen.sh
# dkms build -m zfs -v 2.1.9
# dkms install -m zfs -v 2.1.9
# modprobe zfs
# modinfo zfs | head
filename:       /lib/modules/6.2.8-200.fc37.x86_64/extra/zfs.ko.xz
version:        2.1.9-1
...
#

HTH

openzfs/zfs#14658 (comment)

simonbcn · 2023-04-01T12:25:39Z

In the title of this error dkms has been added but the problem is general: both using dkms and compiling directly without dkms.

On the other hand, if this is fixed but you have not released a new version with this patch, I don't think the issue should be closed. With the current version 2.1.9 this still fails.

When is it planned to release a version with the patch?

adroslice · 2023-04-04T11:05:06Z

For root on zfs users, I highly recommend adding a second kernel and a corresponding bootloader entry to have a fallback for cases exactly like this. It kind of saved me on friday.

Linux 6.3+, and backports from it (6.2.8+), changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rich Ercolani <[email protected]> Closes openzfs#14658 Closes openzfs#14668

castillar · 2023-04-10T03:41:24Z

My home Fedora server has been bitten by this type of issue a couple times in the past — I know it can take ZFS a bit to catch up with kernel releases, so I've tried to be more careful about checking updates and not just blindly slapping them in.

However, to keep from getting surprised, I've also done this and I recommend it to anyone on Fedora looking to make sure this doesn't bite them:

dnf install 'dnf-command(versionlock)'
dnf versionlock add kernel-core kernel-devel kernel-modules

This will lock the kernel packages to the current version and prevent upgrades or removals. Once I've checked to make sure the next kernel rev is safe, I then run dnf versionlock clear, which removes the locks and allows the upgrade. Then I slap it back in place again to put the safety back on the foot-gun. :)

Update the ZFS submodule so that it includes a commit with compat fix [0] for kernel 6.2.8, which otherwise regressed build through the 484c2be84b49 ("block: count 'ios' and 'sectors' when io is done for bio-based device") commit, which was backported to stable-6.2 from the v6.3-rc3 "release". [0]: openzfs/zfs@59f1875 Link: openzfs/zfs#14658 Signed-off-by: Thomas Lamprecht <[email protected]>

Linux 6.3+, and backports from it (6.2.8+), changed the signatures on bdev_io_{start,end}_acct. Add a case for it. Reviewed-by: Brian Behlendorf <[email protected]> Signed-off-by: Rich Ercolani <[email protected]> Closes openzfs#14658 Closes openzfs#14668

comrade-meowski added the Type: Defect Incorrect behavior (e.g. crash, hang) label Mar 23, 2023

RAMChYLD mentioned this issue Mar 23, 2023

zfs git branch (and also 2.1.9 release): include patch 3ace9de and 2e9c9b7 archzfs/archzfs#486

Closed

RaitoBezarius mentioned this issue Mar 24, 2023

zfs: latest compatible kernel is 6.1 rather than 6.2 NixOS/nixpkgs#222942

Closed

12 tasks

RaitoBezarius mentioned this issue Mar 24, 2023

zfsUnstable: make it compatible again with 6.2.8 and potentially 6.3 NixOS/nixpkgs#222946

Merged

12 tasks

RaitoBezarius mentioned this issue Mar 24, 2023

Fixes 5.12 bio bi_bdev additions and GPL issues with bio_*_io_acct functions #11639

Closed

13 tasks

HougeLangley mentioned this issue Mar 26, 2023

Update for 6.2.8 linux and linux-zen packages archzfs/archzfs#487

Closed

szubersk mentioned this issue Mar 27, 2023

ERROR (dkms apport): kernel package linux-headers-6.0.0-6.1-liquorix-amd64 is not supported #14114

Open

testos77 mentioned this issue Mar 27, 2023

Broken due to GPL stevleibelt/arch-linux-live-cd-iso-with-zfs#18

Closed

behlendorf closed this as completed in ae0b1f6 Mar 27, 2023

behlendorf mentioned this issue Mar 27, 2023

[2.1.10] linux 6.3 compat: add another bdev_io_acct case #14677

Merged

13 tasks

AllKind mentioned this issue Mar 28, 2023

modprobe: FATAL: Module zfs not found in directory /lib/modules/6.2.8-200.fc37.x86_64 #14679

Closed

thorstenhirsch mentioned this issue Mar 28, 2023

modprobe btrfs: Key was rejected by service stevleibelt/arch-linux-live-cd-iso-with-zfs#16

Closed

adrian-bl mentioned this issue Mar 28, 2023

ZFS module does not compile with 6.2.8_1 void-linux/void-packages#43013

Closed

mitchejj pushed a commit to mitchejj/ostree-zfs-kmod that referenced this issue Mar 30, 2023

Try to build zfs patch

48f466d

openzfs/zfs#14658 (comment)

stratosgear mentioned this issue Apr 3, 2023

Please expedite release of 2.1.10 #14710

Closed

bsherman mentioned this issue Apr 6, 2023

ZFS 2.1.9 fails to build on testing 37.20230401.2.0 due to Kernel 6.2 ublue-os/ucore#21

Closed

yougotborked mentioned this issue Apr 9, 2023

Fedora 38 zfs-testing repo is not available #14725

Closed

rincebrain mentioned this issue Apr 9, 2023

zfs module doesn't work in latest kernel (6.2.9-100.fc36) and "dnf remove zfs" tries to remove dkms, libvirt etc. #14729

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

6.2.8 kernel revert needed for zfs-2.1.9 dkms #14658

6.2.8 kernel revert needed for zfs-2.1.9 dkms #14658

comrade-meowski commented Mar 23, 2023

philmmanjaro commented Mar 23, 2023

rincebrain commented Mar 23, 2023 •

edited

Loading

simonbcn commented Mar 23, 2023

wtosta commented Mar 23, 2023 •

edited

Loading

ptr1337 commented Mar 23, 2023

HougeLangley commented Mar 23, 2023

demmm commented Mar 23, 2023

rincebrain commented Mar 23, 2023 via email

demmm commented Mar 23, 2023

jonhermansen commented Mar 24, 2023

behlendorf commented Mar 24, 2023

rincebrain commented Mar 24, 2023 via email

rincebrain commented Mar 24, 2023

kode54 commented Mar 26, 2023

ptr1337 commented Mar 26, 2023

imrehg commented Mar 26, 2023

rincebrain commented Mar 26, 2023

lunasophia commented Mar 26, 2023

Alanaktion commented Mar 26, 2023

kode54 commented Mar 26, 2023

lunasophia commented Mar 26, 2023 •

edited

Loading

rincebrain commented Mar 26, 2023

isopix commented Mar 26, 2023 •

edited

Loading

admnd commented Mar 26, 2023 •

edited

Loading

isopix commented Mar 26, 2023 •

edited

Loading

bsdice commented Mar 27, 2023 •

edited

Loading

kode54 commented Mar 27, 2023

youzhongyang commented Mar 27, 2023

andr31g commented Mar 30, 2023

simonbcn commented Apr 1, 2023 •

edited

Loading

adroslice commented Apr 4, 2023

castillar commented Apr 10, 2023

6.2.8 kernel revert needed for zfs-2.1.9 dkms #14658

6.2.8 kernel revert needed for zfs-2.1.9 dkms #14658

Comments

comrade-meowski commented Mar 23, 2023

philmmanjaro commented Mar 23, 2023

rincebrain commented Mar 23, 2023 • edited Loading

simonbcn commented Mar 23, 2023

wtosta commented Mar 23, 2023 • edited Loading

ptr1337 commented Mar 23, 2023

HougeLangley commented Mar 23, 2023

demmm commented Mar 23, 2023

rincebrain commented Mar 23, 2023 via email

demmm commented Mar 23, 2023

jonhermansen commented Mar 24, 2023

behlendorf commented Mar 24, 2023

rincebrain commented Mar 24, 2023 via email

rincebrain commented Mar 24, 2023

kode54 commented Mar 26, 2023

ptr1337 commented Mar 26, 2023

imrehg commented Mar 26, 2023

rincebrain commented Mar 26, 2023

lunasophia commented Mar 26, 2023

Alanaktion commented Mar 26, 2023

kode54 commented Mar 26, 2023

lunasophia commented Mar 26, 2023 • edited Loading

rincebrain commented Mar 26, 2023

isopix commented Mar 26, 2023 • edited Loading

admnd commented Mar 26, 2023 • edited Loading

isopix commented Mar 26, 2023 • edited Loading

bsdice commented Mar 27, 2023 • edited Loading

kode54 commented Mar 27, 2023

youzhongyang commented Mar 27, 2023

andr31g commented Mar 30, 2023

simonbcn commented Apr 1, 2023 • edited Loading

adroslice commented Apr 4, 2023

castillar commented Apr 10, 2023

rincebrain commented Mar 23, 2023 •

edited

Loading

wtosta commented Mar 23, 2023 •

edited

Loading

lunasophia commented Mar 26, 2023 •

edited

Loading

isopix commented Mar 26, 2023 •

edited

Loading

admnd commented Mar 26, 2023 •

edited

Loading

isopix commented Mar 26, 2023 •

edited

Loading

bsdice commented Mar 27, 2023 •

edited

Loading

simonbcn commented Apr 1, 2023 •

edited

Loading