Debian with 2.6.26 : Compilation fails for ZFS #30
Comments
|
Thanks, for the bug report I'll try to work through these issues next week before making an official tag. |
|
Fixed by commit fc3e47c. It looks like there are still a few minor rough edges for Lenny I'll need to work though. In particular the zlib module is called zlib_inflate zlib_deflate, and there are a few more build issues with --enable-debug. |
|
I did a git clone of the complete GIT repo, to have all the latest patches. So now, it compiles (just the problem with the NONE file from SPL). But the zconfig.sh doesn't work. It crashes at the seventh command of the second test. Here is the log (I edited the zconfig.sh to add a 'set -x' just before executing the second test) : splzfs32:/usr/local/libexec/zfs# ./zconfig.sh test 1 - persistent zpool.cache: PASS + zconfig_test2 + POOL_NAME=test2 ++ mktemp + TMP_FILE1=/tmp/tmp.uEPMkLDNGU ++ mktemp + TMP_FILE2=/tmp/tmp.EoKBgpUMJG ++ mktemp -p /tmp zpool.cache.XXXXXXXX + TMP_CACHE=/tmp/zpool.cache.nyBJkbdX + echo -n 'test 2 - scan disks for pools to import: ' test 2 - scan disks for pools to import: + /usr/local/libexec/zfs//zfs.sh zfs=spa_config_path=/tmp/zpool.cache.nyBJkbdX + /usr/local/libexec/zfs//zpool-create.sh -p test2 -c lo-raidz2 + /usr/local/sbin/zpool status test2 + /usr/local/libexec/zfs//zfs.sh -u + rm -f /tmp/zpool.cache.nyBJkbdX + /usr/local/libexec/zfs//zfs.sh zfs=spa_config_path=/tmp/zpool.cache.nyBJkbdX + /usr/local/sbin/zpool import Unable to open /dev/zfs: No such file or directory. Verify the ZFS module stack is loaded by running '/sbin/modprobe zfs'. + grep test2 + fail 7 + echo 'FAIL (7)' FAIL (7) + exit 7 When I try this by hand, it's working. For now, I only tested with 32 bits 2.6.26 kernel. As I'm moving to a new home, I won't have time tomorrow to complete this report. I hope I will have enough time on Monday to complete the report. Also, I will do some tests of a git clone version of spl and zfs under the 2.6.32 kernel on debian stable, in order to confirm it's working. I hope this will be done on next friday. Thanks. |
|
This just looks like an issue with the test scripts. The error your seeing is expected if the ZFS module stack fails to load for some reason. This may be related to the zlib module name changes under Lenny I alluded too in my previous comment. I suspect the scripts just need to be updated detect if your system uses a zlib module or a zlib_inflate/zlib_deflate module. Either way if ZFS was properly installed and depmod run it would not be an issue. |
|
Hi again. This morning, I tried to compile ZFS on 32 and 64 bits 2.6.26 kernels. The compilation works well. But when I try to "modprobe zavl", I get this error : splzfs32:/usr/local/libexec/zfs# modprobe zavl FATAL: Error inserting zavl (/lib/modules/2.6.26-2-686/addon/zfs/avl/zavl.ko): Unknown symbol in module, or unknown parameter (see dmesg) and on 64 bits. splzfs64:/usr/local/libexec# modprobe zavl FATAL: Error inserting zavl (/lib/modules/2.6.26-2-amd64/addon/zfs/avl/zavl.ko): Unknown symbol in module, or unknown parameter (see dmesg) Here is the dmesg : [ 5492.702096] zavl: no symbol version for spl_cleanup [ 5492.702528] zavl: Unknown symbol spl_cleanup [ 5492.702995] zavl: no symbol version for spl_setup [ 5492.703388] zavl: Unknown symbol spl_setup I used a "git clone" version taken this morning to compile it. The SPL compiled well (just the problem with the NONE file, I will try to see this if I have enough time) If you need more info, don't hesitate to ask. I will try on 2.6.32 kernel later today Thanks. |
|
You need to make sure the dependent modules are also loaded. In this case it looks like the spl module was not loaded when you tried to load the zavl module, thus the missing symbols. Run depmod -a on your system and try again this will rebuild the module dependency information for your system. Currently this is done as part of the RPM post install, but if you just did a make install then you'll need to do it manually. If and when we get proper Debian package it can be done there. |
|
Just for information, on the 2.6.32 kernel, it's working well, I just had to add some "sleep 5" on the zconfig.sh script, maybe because the machines I use are not so fast, :). |
|
This is quite strange. I ran the "depmod -a" and a "modprobe zavl", here are the results : splzfs32:~# depmod -a splzfs32:~# modprobe zavl FATAL: Error inserting zavl (/lib/modules/2.6.26-2-686/addon/zfs/avl/zavl.ko): Unknown symbol in module, or unknown parameter (see dmesg) So I tried to load the spl module manually and after, the zavl module : splzfs32:~# modprobe spl splzfs32:~# modprobe zavl FATAL: Error inserting zavl (/lib/modules/2.6.26-2-686/addon/zfs/avl/zavl.ko): Unknown symbol in module, or unknown parameter (see dmesg) The dmesg is still saying the same thing : [ 162.922741] zavl: no symbol version for spl_cleanup [ 162.923217] zavl: Unknown symbol spl_cleanup [ 162.923693] zavl: no symbol version for spl_setup [ 162.924159] zavl: Unknown symbol spl_setup So here, I don't know what's the problem. Maybe the SPL is not compiled correctly, I will double check this tomorrow. Thanks. PS : For the debian packages, I'm already working on it, on the 0.4.9 version, and I set a testing repository which I sent you the address. |
|
Yes, it sounds like a build incompatibility between the spl and zfs. I've seen similar issues when I've accidentally build them against slightly different kernels. |
|
Hi again. And I don't have any good news, :(. I retry to compile SPL and ZFS from a "git clone", and it's still not working. The "modprobe zavl" is still failing with the same errors on the dmesg. I reinstalled a new system for these tests, reinstalled all the build dependencies needed, and here is a log of all the commands I typed : http://paste.davromaniak.eu/pastebin.php?show=m6e652b16 On 64 bits, it's the same scenario, quite strange. Here is the complete extract from the dmesg : http://paste.davromaniak.eu/pastebin.php?show=m74510523 To be honest, I don't know where is the problem, and my knowledge on this type of development is not good enough to help you debugging this. If you need me to type some commands or do some tests, don't hesitate to ask, I'm here to give you many information as you need. Thanks |
|
OK, I've still got a stock Lenny system setup here I'll give it a try and see. |
|
Hmm, I just tested this on my Lenny system and it worked fine. Can you try the following with the latest code, it should load the entire module stack. Oh and you shouldn't need any options for configure any more on Debian systems, it should be able to figure it out now: spl> ./configure && make If this fails can you show me the output of lsmod and dmesg. Thanks. |
|
It's working well. Just for information, here is the lsmod : Module Size Used by zfs 770296 0 zcommon 34736 1 zfs zunicode 319008 1 zfs znvpair 39840 2 zfs,zcommon zavl 5728 1 zfs spl 86104 5 zfs,zcommon,zunicode,znvpair,zavl zlib_deflate 17624 1 zfs zlib_inflate 14144 1 zfs ipv6 235396 10 loop 12748 0 virtio_net 6752 0 virtio_balloon 3972 0 snd_pcsp 8576 0 serio_raw 4740 0 snd_pcm 62660 1 snd_pcsp snd_timer 17800 1 snd_pcm snd 45636 3 snd_pcsp,snd_pcm,snd_timer soundcore 6368 1 snd snd_page_alloc 7816 1 snd_pcm psmouse 32336 0 i2c_piix4 7216 0 i2c_core 19828 1 i2c_piix4 button 6096 0 evdev 8000 0 ext3 105576 1 jbd 39476 1 ext3 mbcache 7108 1 ext3 virtio_blk 4744 3 piix 6568 0 [permanent] floppy 47844 0 virtio_pci 5120 0 ide_pci_generic 3908 0 [permanent] uhci_hcd 18672 0 usbcore 118224 2 uhci_hcd ide_core 96168 2 piix,ide_pci_generic ata_generic 4676 0 libata 140448 1 ata_generic scsi_mod 129548 1 libata dock 8304 1 libata thermal 15228 0 processor 32576 1 thermal fan 4196 0 thermal_sys 10856 3 thermal,processor,fan And thank you for the better support for debian, it's great. It will be more easier for me to create packages for the 0.5.0 version. For the module name problem, maybe you can put an alias in /etc/modprobe.d/zfs for example, this would be less work on SPL/ZFS code I think. Thanks. |
|
OK, good to hear this is working for you now. |
This was referenced Mar 19, 2012
Closed
FransUrbo
pushed a commit
to FransUrbo/zfs
that referenced
this issue
Nov 13, 2013
Allow bootfs on an encrypted filesystem.
akatrevorjay
added a commit
to akatrevorjay/zfs
that referenced
this issue
Dec 16, 2017
# This is the 1st commit message: Merge branch 'master' of https://github.com/zfsonlinux/zfs * 'master' of https://github.com/zfsonlinux/zfs: Enable QAT support in zfs-dkms RPM # This is the commit message openzfs#2: Import 0.6.5.7-0ubuntu3 # This is the commit message openzfs#3: gbp changes # This is the commit message openzfs#4: Bump ver # This is the commit message openzfs#5: -j9 baby # This is the commit message openzfs#6: Up # This is the commit message openzfs#7: Yup # This is the commit message openzfs#8: Add new module # This is the commit message openzfs#9: Up # This is the commit message openzfs#10: Up # This is the commit message openzfs#11: Bump # This is the commit message openzfs#12: Grr # This is the commit message openzfs#13: Yay # This is the commit message openzfs#14: Yay # This is the commit message openzfs#15: Yay # This is the commit message openzfs#16: Yay # This is the commit message openzfs#17: Yay # This is the commit message openzfs#18: Yay # This is the commit message openzfs#19: yay # This is the commit message openzfs#20: yay # This is the commit message openzfs#21: yay # This is the commit message openzfs#22: Update ppa script # This is the commit message openzfs#23: Update gbp conf with br changes # This is the commit message openzfs#24: Update gbp conf with br changes # This is the commit message openzfs#25: Bump # This is the commit message openzfs#26: No pristine # This is the commit message openzfs#27: Bump # This is the commit message openzfs#28: Lol whoops # This is the commit message openzfs#29: Fix name # This is the commit message openzfs#30: Fix name # This is the commit message openzfs#31: rebase # This is the commit message openzfs#32: Bump # This is the commit message openzfs#33: Bump # This is the commit message openzfs#34: Bump # This is the commit message openzfs#35: Bump # This is the commit message openzfs#36: ntrim # This is the commit message openzfs#37: Bump # This is the commit message openzfs#38: 9 # This is the commit message openzfs#39: Bump # This is the commit message openzfs#40: Bump # This is the commit message openzfs#41: Bump # This is the commit message openzfs#42: Revert "9" This reverts commit de488f1. # This is the commit message openzfs#43: Bump # This is the commit message openzfs#44: Account for zconfig.sh being removed # This is the commit message openzfs#45: Bump # This is the commit message openzfs#46: Add artful # This is the commit message openzfs#47: Add in zed.d and zpool.d scripts # This is the commit message openzfs#48: Bump # This is the commit message openzfs#49: Bump # This is the commit message openzfs#50: Bump # This is the commit message openzfs#51: Bump # This is the commit message openzfs#52: ugh # This is the commit message openzfs#53: fix zed upgrade # This is the commit message openzfs#54: Bump # This is the commit message openzfs#55: conf file zed.d # This is the commit message #56: Bump
12 tasks
12 tasks
Closed
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
May 24, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
May 24, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
May 24, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
May 28, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
May 31, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
Jun 6, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time. macOS: more work Upstream: zfs_log can't VN_HOLD a possibly unlinked vp Follow in FreeBSD steps, and avoid the first call to VN_HOLD in case it is unlinked, as that can deadlock waiting in vnode_iocount(). Walk up the xattr_parent.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
Jun 6, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time. macOS: more work Upstream: zfs_log can't VN_HOLD a possibly unlinked vp Follow in FreeBSD steps, and avoid the first call to VN_HOLD in case it is unlinked, as that can deadlock waiting in vnode_iocount(). Walk up the xattr_parent.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
Jun 6, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time. macOS: more work Upstream: zfs_log can't VN_HOLD a possibly unlinked vp Follow in FreeBSD steps, and avoid the first call to VN_HOLD in case it is unlinked, as that can deadlock waiting in vnode_iocount(). Walk up the xattr_parent.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
Jun 6, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time. macOS: more work Upstream: zfs_log can't VN_HOLD a possibly unlinked vp Follow in FreeBSD steps, and avoid the first call to VN_HOLD in case it is unlinked, as that can deadlock waiting in vnode_iocount(). Walk up the xattr_parent.
lundman
added a commit
to openzfsonosx/openzfs-fork
that referenced
this issue
Jun 8, 2021
Add all files required for the macOS port. Add new cmd/os/ for tools which are only expected to be used on macOS. This has support for all macOS version up to Catalina. (Not BigSur). Signed-off-by: Jorgen Lundman <[email protected]> macOS: big uio change over. Make uio be internal (ZFS) struct, possibly referring to supplied (XNU) uio from kernel. This means zio_crypto.c can now be identical to upstream. Update for draid, and other changes macOS: Use SET_ERROR with uiomove. [squash] macOS: they went and added vdev_draid macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too Upstream: avoid warning zio_crypt.c:1302:3: warning: passing 'const struct iovec *' to parameter of type 'void *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers] kmem_free(uio->uio_iov, uio->uio_iovcnt * sizeof (iovec_t)); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ macOS: Update zfs_acl.c to latest This includes commits like: 65c7cc4 1b376d1 cfdc432 716b53d a741b38 485b50b macOS: struct vdev changes macOS: cstyle, how you vex me [squash] Upstream: booo Werror booo Upstream: squash baby Not defined gives warnings. Upstream: Include all Makefiles Signed-off-by: Jorgen Lundman <[email protected]> double draid! macOS: large commit macOS: Use APPLE approved kmem_alloc() macOS: large commit WIP: remove reliance on zfs.exports The memory-pressure has been nerfed, and will not run well until we can find other solutions. The kext symbol lookup we can live without, used only for debug and panic. Use lldb to lookup symbols. leaner! leanerr! remove zfs.export dependency cont. export reduction cont. cont. Corrective tweaks for building Correct vnode_iocount() Cleanup pipe wrap code, use pthreads, handle multiple streams latest pipe send with threads sort of works, but bad timing can be deadlock macOS: work out corner case starvation issue in cv_wait_sig() Fix -C in zfs send/recv cv_wait_sig squash Also wrap zfs send resume Implement VOP_LOOKUP for snowflake Finder Don't change date when setting size. Seems to be a weird required with linux, so model after freebsd version macOS: correct xattr checks for uio Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures Fix a noisy source of misleading-indentation warnings Fix "make install" ln -s failures fix ASSERT: don't try to peer into opaque vp structure Import non-panicking ASSERT from old spl/include/sys/debug.h Guard with MACOS_ASSERT_SHOULD_PANIC which will do what Linux and FreeBSD do: redefine ASSERTs as VERIFYs. The panic report line will say VERIFY obscuring the problem, and a system panic is harsher (and more dangerous) on MacOS than a zfs-module panic on Linux. ASSERTions: declare assfail in debug.h Build and link spl-debug.c Eliminate spurious "off" variable, use position+offset range Make sure we hold the correct range to avoid panic in dmu_tx_dirty_buf (macro DMU_TX_DIRTY_BUF defined with --enable-debug). zvol_log_write the range we have written, not the future range silence very noisy and dubious ASSERT macOS: M1 fixes for arm64. sysctl needs to use OID2 Allocs needs to be IOMalloc_aligned Initial spl-vmem memory area needs to be aligned to 16KB No cpu_number() for arm64. macOS: change zvol locking, add zvol symlinks macOS: Return error on UF_COMPRESSED This means bsdtar will be rather noisy, but we prefer noise over corrupt files (all files would be 0-sized). usr/bin/zprint: Failed to set file flags~ -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint usr/bin/zprint: Failed to set file flags -rwxr-xr-x 1 root wheel 47024 Mar 17 2020 /Volumes/BOOM/usr/bin/zprint Actually include zedlet for zvols macOS: Fix Finder crash on quickview, SMB error codes xattr=sa would return negative returncode, hangover from ZOL code. Only set size if passed a ptr. Convert negative errors codes back to normal. Add LIBTOOLFLAGS for macports toolchain This will replace PR#23 macOS zpool import fixes The new codebase uses a mixture of thread pools and lio_listio async io, and on macOS there are low aio limits, and when those are reached lio_listio() returns EAGAIN when probing several prospective leaf vdevs concurrently, looking for labels. We should not abandon probing a vdev in this case, and can usually recover by trying again after a short delay. (We continue to treat other errnos as unrecoverable for that vdev, and only try to recover from EAGAIN a few times). Additionally, take logic from old o3x and don't probe a variety of devices commonly found in /dev/XXX as they either produce side-effects or are simply wasted effort. Finally, add a trailing / that FreeBSD and Linux both have. listxattr may not expose com.apple.system xattr=sa We need to ask IOMallocAligned for the enclosing POW2 vmem_create() arenas want at least natural alignment for the spans they import, and will panic if they don't get it. For sub-PAGESIZE calls to osif_malloc, align on PAGESIZE. Otherwise align on the enclosing power of two for any osif_malloc allocation up to 2^32. Anything that asks osif_malloc() for more than that is almost certainly a bug, but we can try aligning on PAGESIZE anyway, rather than extend the enclosing-power-of-two device to handle 64-bit allocations. Simplify the creation of bucket arenas, and adjust their quanta. This results in handing back considerably more (and smaller) chunks of memory to osif_free if there is pressure, and reduces waits in xnu_alloc_throttled(), so is a performance win for a busy memory-constrained system. Finally, uncomment some valid code that might be used by future callers of vmem_xcreate(). use vmem_xalloc to match the vmem_xfree of initial dynamic alloc vmem_alloc() breaks the initial large vmem_add() allocation into smaller chunks in an effort to have a large number vmem segments in the arena. This arena does not benefit from that. Additionaly, in vmem_fini() we call vmem_xfree() to return the initial allocation because it is done after almost everything has been pulled down. Unfortunately vmem_xfree() returns the entire initial allocation as a single span. IOFree() checks a variable maintained by the IOMalloc* allocators which tracks the largest allocation made so far, and will panic when (as it almost always is the case) the initial large span is handed to it. This usually manifests as a panic or hang on kext unload, or a hang at reboot. Consequently, we will now use vmem_xalloc() for this initial allocation; vmem_xalloc() also lets us explicitly specify the natural alignement we want for it. zfs_rename SA_ADDTIME may grow SA Avoid: zfs`dmu_tx_dirty_buf(tx=0xffffff8890a56e40, db=0xffffff8890ae8cd0) at dmu_tx.c:674:2 -> 674 panic("dirtying dbuf obj=%llx lvl=%u blkid=%llx but not tx_held\n", 675 (u_longlong_t)db->db.db_object, db->db_level, 676 (u_longlong_t)db->db_blkid); zfs diff also needs to be wrapped. Replace call to pipe() with a couple of open(mkfifo) instead. Upstream: cstyle zfs_fm.c macOS: cstyle baby IOMallocAligned() should call IOFreeAligned() macOS: zpool_disable_volumes v1 When exporting, also kick mounted zvols offline macOS: zpool_disable_volumes v2 When exporting zvols, check IOReg for the BSDName, instead of using readlink on the ZVOL symlinks. Also check if apfs has made any synthesized disks, and ask them to unmount first. ./scripts/cmd-macos.sh zpool export BOOM Exporting 'BOOM/volume' ... asking apfs to eject 'disk5' Unmount of all volumes on disk5 was successful ... asking apfs to eject 'disk5s1' Unmount of all volumes on disk5 was successful ... asking ZVOL to export 'disk4' Unmount of all volumes on disk4 was successful zpool_disable_volume: exit macOS: Add libdiskmgt and call inuse checks macOS: compile fixes from rebase macOS: oh cstyle, how you vex me so macOS: They added new methods - squash macOS: arc_register_hotplug for userland too macOS: minor tweaks for libdiskmgt macOS: getxattr size==0 is to lookup size Also skip the ENOENT return for "zero" finderinfo, as we do not skip over them in listxattr. macOS: 10.9 compile fixes macOS: go to rc2 macOS: kstat string handling should copyin. cstyle baby macOS: Initialise ALL quota types projectid, userobj, groupobj and projectobj, quotas were missed. macOS: error check sysctl for older macOS Wooo cstyle, \o/ Make arc sysctl tunables work (#27) * use an IOMemAligned for a PAGE_SIZE allocation * we should call arc_kstat_update_osx() Changing kstat.zfs.darwin.tunable.zfs_arc_min doesn't do anything becasue arc_kstat_update_osx() was removed at the same time the (obsoleted by upstream) arc_kstat_update() was removed from zfs_kstat_osx.c. Put it back. * when we sysctl arc tunables, call arc_tuning_update() * rely on upstream's sanity checking Simplification which also avoids spurious CMN_WARN messages caused by setting the arcstat variables here, when upstream's arc_tuning_update() checks that they differ from the tunable variables. * add tunable zfs_arc_sys_free and fix zfs_arc_lotsfree_percent both are in upstream's arc_tuning_update() zfs_arc_sys_free controls the amount of memory that ARC will leave free, which is roughly what lundman wants for putting some sort of cap on memory use. * cstyle macOS: set UIO direction, to receive xattr from XNU macOS: ensure uio is zeroed in case XNU uio is NULL. Fix zfs_vnop_getxattr (openzfs#28) "xattr -l <file>" would return inconsistent garbage, especially from non-com.apple.FinderInfo xattrs. The UIO_WRITE was a 0 (UIO_READ) in the previous commit, change it. Also turn kmem_alloc -> kmem_zalloc in zfs_vnops_osx.c, for cheap extra safety. launch `zpool import` through launchd in the startup script (#26) Signed-off-by: Guillaume Lessard <[email protected]> cstyle macOS: correct dataset_kstat_ logic and kstat leak. dataset_kstat_create() will allocate a string and set it before calling kstat_create() - so we can not set strings to NULL. Likewise, we can not bulk free strings on unload, we have to rely on the caller of kstat to do so. (Which is proper). Add calls to dataset_kstat for datasets and zvol. kstat.zfs/BOOM.dataset.objset-0x36.dataset_name: BOOM kstat.zfs/BOOM.dataset.objset-0x36.writes: 0 kstat.zfs/BOOM.dataset.objset-0x36.nwritten: 0 kstat.zfs/BOOM.dataset.objset-0x36.reads: 11 kstat.zfs/BOOM.dataset.objset-0x36.nread: 10810 kstat.zfs/BOOM.dataset.objset-0x36.nunlinks: 0 kstat.zfs/BOOM.dataset.objset-0x36.nunlinked: 0 macOS: remove no previous prototype for function macOS: correct openat wrapper build fixes re TargetConditionals.h (openzfs#30) AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Memory fixes on macOS_pure (openzfs#31) * Improve memory handling on macOS * remove obsolete/unused zfs_file_data/zfs_metadata caching * In the new code base, we use upstream's zio.c without modification, and so the special zio caching code became entirely vestigial, and likely counterproductive. * and make busy ABD better behaved on busy macOS box Post-ABD we no longer gained much benefit in the old code base from the complicated special handling for the caches created in zio.c. As there's only really one size of ABD allocation, we do not need a qcache layer as in 1.9. Instead use an arena with VMC_NO_QCACHE set to ask for for 256k chunks. * don't reap extra caches in arc_kmem_reap_now() KMF_LITE in DEBUG build is OK * build fixes re TargetConditionals.h AvailabilityMacros.h needs TargetConditionals.h defintions in picky modern compilers. Add them to sysmacros.h, and fix a missing sysmacros.h include. Use barrier synchronization and IO Priority in ldi_iokit.cpp (openzfs#33) * other minor changes in vdev_disk Thread and taskq fixing (openzfs#32) Highlights: * thread names for spindump * some taskq_d is safe and useful * reduce thread priorities * use througput & latency QOS * TIMESHARE scheduling * passivate some IO * Pull in relevant changes from old taskq_fixing branch 1.9 experimentation pulled into 2.x * add throttle_set_thread_io_policy to zfs.exports * selectively re-enable TASKQ_DYNAMIC also drop wr_iss zio taskqs even further in priority (cf freebsd) * reduce zvol taskq priority * make system_taskq dynamic * experimentally allow three more taskq_d * lower thread prorities overall on an M1 with no zfs whatsoever, the highest priority threads are in the mid 90s, with most kernel threads at priority 81 (basepri). with so many maxclsyspri threads in zfs, we owuld starve out important things like vm_pageout_scan (pri 91), sched_maintenance_thread (pri 95), and numerous others. moreover, ifnet_start_{interfaces} are all priority 82. we should drop minclsyspri below 81, have defclsyspri at no more than 81, and make sure we have few threads above 89. * some tidying up of lowering of priority Thread and taskq fixing * fix old code pulled into spa.c, and further lower priorities * Thread and taskq fixing drop xnu priorities by one update a comment block set USER_INITIATED throughput QOS on TIMESHARE taskq threads don't boost taskq threads accidentally don't let taskq threads be pri==81 don't let o3x threads have importance > 0 apply xnu thread policies to taskq_d threads too assuming this works, it calls out for DRY refactoring with the other two flavours, that operate on current_thread(). simplify in spa.c make practically all the taskqs TIMESHARE Revert "apply xnu thread policies to taskq_d threads too" Panic in VM This reverts commit 39f93be. Revert "Revert "apply xnu thread policies to taskq_d threads too"" I see what happened now. This reverts commit 75619f0. adjust thread not the magic number refactor setting thread qos make DRY refactor rebuild this includes userland TASKQ_REALLY_DYNAMIC fixes fix typo set thread names for spindump visibility cstyle Upstream: Add --enable-macos-impure to autoconf Controls -DMACOS_IMPURE Signed-off-by: Jorgen lundman <[email protected]> macOS: Add --enable-macos-impure switch to missing calls. Call the wrapped spl_throttle_set_thread_io_policy Add spl_throttle_set_thread_io_policy to headers macOS: vdev_file should use file_taskq Also cleanup spl-taskq to have taskq_wait_outstanding() in preparation for one day implementing it. Change alloc to zalloc in zfs_ctldir.c Call wrap_zstd_init() and wrap_zstd_fini() (openzfs#34) macOS: change both alloc to zalloc macOS: mutex_tryenter can be used while holding zstd uses mutex_tryenter() to check if it already is holding the mutex. Can't find any implementations that object to it, so changing our spl-mutex.c Tag zfs-2.0.0rc4 macOS: return error from uiomove instead of panic macOS: Skip known /dev entry which hangs macOS: Give better error msg when features are needed for crypto Using 1.9.4 crypto dataset now require userobj and projectquota. Alert the user to activate said features to mount crypt dataset. There is no going back to 1.9.4 after features are enabled. macOS: Revert to pread() over AIO due to platform issues. We see waves of EAGAIN errors from lio_listio() on BigSur (but not Catalina) which could stem from recent changes to AIO in XNU. For now, we will go with the classic read label. Re-introduce a purified memory pressure handling mechanism (openzfs#35) * Introduce pure pressure-detecting-and-reacting system * "pure" -- no zfs.exports requirement * plumb in mach_vm_pressure_level_monitor() and mach_vm_pressure_monitor() calls to maintain reduced set of inputs into previous signalling into (increasingly shared with upstream) arc growth or shrinking policy * introduce mach_vm_pressure kstats which can be compared with userland-only sysctls: kstat.spl.misc.spl_misc.spl_vm_pages_reclaimed: 0 kstat.spl.misc.spl_misc.spl_vm_pages_wanted: 0 kstat.spl.misc.spl_misc.spl_vm_pressure_level: 0 vm.page_free_wanted: 0 vm.page_free_count: 25,545 vm.page_speculative_count: 148,572 * and a start on tidying and obsolete code elimination * make arc_default_max() much bigger Optional: can be squashed into main pressure commit, or omitted. Users can use zsysctl.conf or manual setting of kstat.zfs.darwin.tunable.zfs_arc_max to override whichever default is chosen (this one, or the one it replaces). Allmem is already deflated during initialization, so this patch raises the un-sysctled ARC maximum from 1/6 to 1/2 of physmem. * handle (vmem) abd_cache fragmentation after arc shrink When arc shrinks due to a significant pressure event, the abd_chunk kmem cache will free slabs back to the vmem abd_cache, and this memory can be several gigabytes. Unfortunately multi-threaded concurrent kmem_cache allocation in the first place, and a priori unpredicatble arc object lifetimes means that abds held by arc objects may be scattered across multiple slabs, with different objects interleaved within slabs. Thus after a moderate free, the vmem cache can be fragmented and this is seen by (sysctl) kstat.vmem.vmem.abd_cache.mem_inuse being much smaller than (sysctl) kstat.vmem.vmem.abd_cache.mem_import, the latter of which may even be stuck at approximately the same value as before the arc free and kmem_cache reap. When there is a large difference between import and inuse, we set arc_no_grow in hopes that ongoing arc activity will defragment organically. This works better with more arc read/write activity after the free, and almost not at all if after the free there is almost no activity. We also add BESTFIT policy to abd_arena experimentally BESTFIT: look harder to place an abd chunk in a slab rather than place in the first slot that is definitely large enough which breaks the vmem constant-time allocation guarantee, although that is less important for this particular vmem arena because of the strong modality of allocations from the abd_chunk cache (its only client). Additionally reduce the abd_cache arena import size to 128k from 256k; the increase in allocation and free traffic between it and the heap is small compared to the gain under this new anti-fragmentation scheme. * some additional tidying in arc_os.c Tag macos-2.0.0-rc5 abd_cache fragmentation mitigation (openzfs#36) * printf->dprintf HFS_GET_BOOT_INFO periodically there will be huge numbers of these printfs, and they are not really useful except when debugging vnops. * Mitigate fragmentation in vmem.abd_cache In macOS_pure the abd_chunk kmem cache is parented to the abd_cache vmem arena to avoid sometimes-heavy ARC allocation and free stress on the main kmem cache, and because abd_chunk has such a strongly modal page-sized allocation size. Additionally, abd_chunk allocations and frees come in gangs, often with high multi-thread concurrency. It is that latter property which is the primary source of arena fragmentation, and it will affect any vmem arena directly underneath the abd_chunk kmem cache. Because we have a vmeme parent solely for abd_chunk, we can monitor that parent for various patterns and react to them. This patch monitors the difference between the variables exported as kstat.vmem.vmem.abd_cache.mem_inuse and kstat.vmem.vmem.abd_cache.mem_import, watching for a large gap between the two, which can arise after an ARC shrink returns many slabs from the arc_chunk kmem cache to the abd_cache arena, as vmem segments still contain slabs which hold still-alive abds. When there is a significant gap, we turn on arc_no_grow and hope that organic ARC activity reduces the gap. If after several minutes this is not the case, a small arc_reduce_target_size() is applied. In comparison with previous behaviour, ARC equilibrium sizes will tend slightly -- but not neormously -- lower because the arc target size reduction is made fairly frequently. However, this is offset by the benefit of less *long-term* abd_cache fragmentation, and less complete collapses of ARC in the face of system memory pressure (since less is "stuck" in vmem). ARC consequently will stay at its equilibrium more often than near its minimum. This is demonstrated by a generally lower overall total held memory (kstat.spl.misc.spl_misc.os_mem_alloc) except on systems with essentially no memory pressure, or systems which have been sysctl-tuned for different behaviour. macOS: Additional 10.9 fixes that missed the boat Tidying nvram zfs_boot=pool (openzfs#37) If zfs_boot is set we run a long-lived zfs_boot_import_thread, which can stay running until the kernel module is running _fini() functions at unload or shutdown. This patch dispatches it on a zfs_boot() taskq, to avoid causing a hang at the taskq_wait_outstanding(system_taskq, 0) in zvol.c's zvol_create_minors_recursive(), which would prevent pool imports finishing if the pool contained zvols. (Symptoms: "zpool import" does not exit for any pool, system does not see any zvols). This exposed a long-term race condition in our zfs_boot.cpp: the notifier can cause the mutex_enter(&pools->lock) in zfs_boot_probe_media to be reached before the mutex_enter() after the notifier was created. The use of the system_taskq was masking that, by quietly imposing a serialization choke. Moving the mutex and cv initialization earlier -- in particular before the notifier is created -- eliminates the race. Further tidying in zfs_boot.cpp, including some cstyling, switching to _Atomic instead of volatile. Volatile is for effectively random reads; _Atomic is for when we want many readers to have a consistent view after the variable is written. Finally, we need TargetConditionals.h in front of AvailabilityMacros.h in order to build. Add includes to build on Big Sur with macports-clang-11 (openzfs#38) * TargetConditionals.h before all AvailabilityMacros.h * add several TargetConditionals.h and AvaialbilityMacros.h Satisfy picky macports-clang-11 toolchain on Big Sur. macOS: clean up large build, indicate errors. Fix debug macOS: Retire MNTTYPE_ZFS_SUBTYPE lookup zfs in iokit macOS: rename net.lundman. -> org.openzfsonosx. macOS: Tag va_mode for upstream ASSERTS XNU sets va_type = VDIR, but does not bother with va_mode. However ZFS checks to confirm S_ISDIR is set in mkdir. macOS: Fix zfs_ioc_osx_proxy_dataset for datasets It was defined as a _pool() ioctl. While we are here changing things change it into a new-style ioctl instead. This should fix non-root datasets mounting as a proxy (devdisk=on). cstyle macOS: setxattr debug prints left in macOS: don't create DYNAMIC with _ent taskq macOS: Also uninstall new /usr/local/zfs before install macos-2.0.0-rc6 macOS: strcmp deprecated after macOS 11 macOS: pkg needs to notarize at the end macOS: strdup strings in getmntent mirrored on FreeBSD. macOS: remove debug print macOS: unload zfs, not openzfs macOS: actually include the volume icon file as well also update to PR macOS: prefer disk over rdisk macOS: devdisk=off mimic=on needs to check for dataset Datasets with devdisks=on will be in ioreg, with it off and mimic=on then it needs to handle: BOOM/fs1 /Volumes/BOOM/fs1 by testing if "BOOM/fs1" is a valid dataset. fixifx macOS: doubled up "int rc" losing returncode Causing misleading messages macOS: zfsctl was sending from IDs macOS: let zfs mount as user succeed If the "mkdir" can succeed (home dir etc, as opposed to /Volumes) then let the mount be able to happen. macOS: Attempt to implement taskq_dispatch_delay() frequently used with taskq_cancel_id() to stop taskq from calling `func()` before the timeout expires. Currently implemented by the taskq sleeping in cv_timedwait() until timeout expires, or it is signalled by taskq_cancel_id(). Seems a little undesirable, could we build an ordered list of delayed taskqs, and only place them to run once timeout has expired, leaving the taskq available to work instead of delaying. macOS: Separate unmount and proxy_remove When proxy_remove is called at the tail end of unmount, we get the alert about "ejecting before disconnecting device". To mirror the proxy create, we make it a separate ioctl, and issue it after unmount completes. macOS: explicitly call setsize with O_TRUNC It appears O_TRUNC does nothing, like the goggles. macOS: Add O_APPEND to zfs_file_t It is currently not used, but since it was written for a test case, we might as well keep it. macOS: Pass fd_offset between kernel and userland. macOS: Missing return in non-void function macOS: finally fix taskq_dispatch_delay() you find a bug, you own the bug. macOS: add missing kstats macOS: restore the default system_delay_taskq macOS: dont call taskq_wait in taskq_cancel macOS: fix taskq_cancel_id() We need to make sure the taskq has finished before returning in taskq_cancel_id(), so that the taskq doesn't get a chance to run after. macOS: correct 'hz' to 100. sysctl kern.clockrate: 100 sleeping for 1 second. bolt: 681571 sleep() 35 bolt: 681672: diff 101 'hz' is definitely 100. macOS: implement taskq_delay_dispatch() Implement delayed taskq by adding them to a list, sorted by wake-up time, and a dispatcher thread which sleeps until the soonest taskq is due. taskq_cancel_id() will remove task from list if present. macOS: ensure to use 1024 version of struct statfs and avoid coredump if passed zhp == NULL. macOS: fix memory leak in xattr_list macOS: must define D__DARWIN_64_BIT_INO_T for working getfsstat getmntany: don't set _DARWIN_FEATURE_64_BIT_INODE This is automatically set by default in userland if the deployment target is > 10.5 macOS: Fix watchdog unload and delay() macOS: improve handling of invariant disks Don't prepend /dev to all paths not starting with /dev as InvariantDisks places its symlinks in /var/run/disk/by-* not /dev/disk/by-*. Also, merge in some tweaks from Linux's zpool_vdev_os.c such as only using O_EXCL with spares. macOS: remove zfs_unmount_006_pos from large. Results in KILLED. Tag macos-2.0.0rc7 macOS: If we don't set SOURCES it makes up zfs.c from nowhere macOS: remove warning macOS: compile fixes after rebase macOS: connect SEEK_HOLE SEEK_DATA to ioctl macOS: Only call vnode_specrdev() when valid macOS: Use VNODE_RELOAD in iterate in the hopes of avoiding ZFS call back in VNOP_INACTIVE macOS: zfs_kmod_fini() calls taskq_cancel_id() so we must unload system_taskq_fini() after the call to zfs_kmod_fini() macOS: shellcheck error macOS: Setting landmines cause panic on M1 "panicString" : "panic(cpu 1 caller 0xfffffe001db72dc8): Break 0xC470 instruction exception from kernel. Ptrauth failure with IA key resulted in 0x2000000000000001 at pc 0xfffffe001c630880, lr 0x8afcfe001c630864 (saved state: 0xfffffe309386b180) macOS: vget should only lookup direct IDs macOS: rootzp left z_projid uninitialised Causing z_projid to have "0xBADDCAFEBADDCAFE" initially, and zfs_link() to return EXDEV due to differenting z_projid, presenting the user with "Cross-device link". Would only happen after loading kext, on the root znode. macOS: Update installer rtf macOS: update and correct the kext_version macOS: Update copyright, fix url and versions macOS ARC memory improvements and old code removal macOS_pure "purification" in spl-[kv]mem coupled with the new dynamics of trying to contain the split between inuse and allocated in the ABD vmem arena produce less memory-greed, so we don't have to do as much policing of memory consumption, and lets us rely on some more common/cross-platform code for a number of commonplace calculation and adjustment of ARC variables. Additionally: * Greater niceness in spl_free_thread : when we see pages are wanted (but no xnu pressure), react more strongly. Notably if we are within 64MB of zfs's memory ceiling, clamp spl_free to a maximum of 32MB. * following recent fixes to abd_os.c, revert to KMC_NOTOUCH at abd_chunk kmem cache creation time, to turn off BUFTAG|CONTENTS|LITE, thus avoiding allocations of many many extra 4k chunks in DEBUG builds. * Double prepopulation of kmem_taskq entries: kmem_cache_applyall() makes this busy, and we want at least as many entries as we have kmem caches at kmem_reqp() time. macOS: more work Upstream: zfs_log can't VN_HOLD a possibly unlinked vp Follow in FreeBSD steps, and avoid the first call to VN_HOLD in case it is unlinked, as that can deadlock waiting in vnode_iocount(). Walk up the xattr_parent.
tonynguien
pushed a commit
to tonynguien/zfs
that referenced
this issue
Dec 21, 2021
This issue was closed.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi.
After successfully compile and install SPL (thanks to the issue 14 from SPL : http://github.com/behlendorf/spl/issues/closed#issue/14 ), and creating the NONE file by hand.
The compilation fails.
The full version is here : http://paste.davromaniak.eu/pastebin.php?show=m6ddd2e2
It's the same problem on 32 and 64 bits arch.
Thanks.
The text was updated successfully, but these errors were encountered: