Add macOS support to OpenZFS #12110

lundman · 2021-05-24T06:49:58Z

Motivation and Context

Add macOS support.

Yes, one giant commit - but don't panic.

At first; the easy, independent, changes were taken out, and individually PRed against ZOL. Now we've reached a point where there is not so many of those opportunities left.

If you spot a change you would prefer to be PRed separately, please let me know.

What's in the giant commit?

macOS specific files, C, C++, and obj-C source files with headers.
automake and build environment, Makefile changes.
changes needed in common source files, with an attempt to minimize these.

Changes in tests/ have been left out, and will be in some future PR. Current zfs-tests pass-rate is about 50%. This is mostly due to wrong Unix tools used etc, as opposed to failure to perform ZFS tasks.

Interesting points of ... interests...

configure uses sed -r, changed to $GSED, but could also handle sed -E.
Linux squats on mount_zfs, some rename work added to macos/mount_zfs. cmd/os/linux/mount_zfs ?
5 new ZFS_PROP
2 new ZFS ioctl
2 new zpl_attr
struct spa, SPA_OS_FIELDS better?
ZIO_OS_FIELDS
Separate assembler files. Should use ENTRY or similar macro, and defined list of scratch regs
zfs send/recv are always through unix domain pipes. (libzfs_macos_pipefd())
separate zpool_read_label, but I think ZOL now has 2 paths, avoiding broken lio
does make clean work? make install -> any macOS files on linux/freebsd?

Description

This adds support for macOS 10.9 to macOS 12 including intel and arm64.

How Has This Been Tested?

macOS zfs-tester and numerous unfortunate users.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Performance enhancement (non-breaking change which improves efficiency)
Code cleanup (non-breaking change which makes code smaller or more readable)
Breaking change (fix or feature that would cause existing functionality to change)
Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
Documentation (a change to man pages or other documentation)

Checklist:

My code follows the OpenZFS code style requirements.
I have updated the documentation accordingly.
I have read the contributing document.
I have added tests to cover my changes.
I have run the ZFS Test Suite with this change applied.
All commit messages are properly formatted and contain Signed-off-by.

lundman · 2023-07-19T10:30:38Z

This PR has been updated up to Sonoma (Apple clang version 15.0.0 (clang-1500.0.34.3)), x86_64 and arm64.
All warnings removed, which unfortunately meant fiddling inside shared sources some more.
If it is preferred any specific change to be pulled out in separate PR, that can be done.
arm64 can be unified instead of #ifdef when they are no longer updated.

andrewc12 · 2023-10-20T07:29:11Z

@behlendorf @lundman @mcmilk I remember that we were waiting until the next release to look at this. Since 2.2 has released is there a chance of getting this reviewed for merging?

I also remember someone brought up the ideas of

mostly reviewing the shared code, and
initially dev work not blocking on macos problems (with lundman fixing things)

mcmilk · 2023-10-22T07:14:57Z

It would be nice when this PR gets into current master. I would help afterwards with the unified asm_linkage.h ;-)

mcmilk

We need to split this big commit into smaller parts.

At least ito these 4 commits:
part1: module/os/macos/* + include/os/macos/*
part2: module/*
part3: cmd/* + tests/*
part4: config/, scripts/, contrib/* and all other things
-> If you want, I can also create an example of that.

It shouldn't interfare the current CI, so that testing other PRs is working...
Maybe @behlendorf has also some ideas what needs to be done before merging.

lundman · 2023-10-23T00:17:05Z

Couple of bonus ones in there with 5 and 6. Let me know if you want them squashed into 1-4 somewhere.

andrewc12 · 2023-10-24T05:55:31Z

I've broken out the commits into separate pull requests so I can run the tests against them
#15440
#15441
#15442
#15443
#15444
#15445

2-6
#15446

andrewc12 · 2023-10-24T06:20:03Z

@lundman I tried compiling everything without the macos code (2-6)
It complained about

automake: error: cannot open < module/os/macos/Makefile.am: No such file or directory

so if this PR ends up getting split into multiple PRs that technically counts as shared code

lundman · 2023-10-24T06:46:27Z

Yep, all Makefiles are now read in all at once - something upstream did a couple of years back, and some of those Makefiles do various things, depending on .in files etc. The Makefile changes have to be last and in one go.

part1: module/os/macos/* + include/os/macos/* This does then inlcude lib/*macos* as well, using grep so there are outliers, like manpage. Signed-off-by: Jorgen Lundman <[email protected]>

part2: module/* Signed-off-by: Jorgen Lundman <[email protected]>

part3: cmd/* + tests/* Signed-off-by: Jorgen Lundman <[email protected]>

part4: config/, scripts/, contrib/* Signed-off-by: Jorgen Lundman <[email protected]>

Bonus part5, the non-macos changes to include lib and the top level Makefiles Signed-off-by: Jorgen Lundman <[email protected]>

Signed-off-by: Jorgen Lundman <[email protected]>

yurikoles · 2023-11-05T02:51:54Z

.github/workflows/macos-build.yaml

+          ./configure CPPFLAGS="-I/usr/local/opt/gettext/include -I/usr/local/opt/openssl/include" LDFLAGS="-L/usr/local/opt/gettext/lib/ -L/usr/local/opt/openssl/lib"
+      - name: build
+        run: |
+          make -j 2


The default GitHub-hosted macOS runner has 3 vCPUs, but you are limiting make to only 2 jobs.

This then?

make -j`nproc`

Or

make -j$((`nproc`+1))

nproc is part of GNU Core Utilities, but macOS ships with CLI tools from BSD. A native macOS analogue is sysctl -n hw.ncpu, there is also a hw.physicalcpu and a hw.logicalcpu for hyper-threading.

make -j `sysctl -n hw.ncpu` make -j $((`sysctl -n hw.ncpu`+1))

Signed-off-by: Andrew Innes <[email protected]>

Signed-off-by: Yurii Kolesnykov <[email protected]>

Signed-off-by: Andrew Innes <[email protected]>

andrewc12 · 2024-01-17T08:24:15Z

#15784

Ultimately, I think you can proceed with Linux/FreeBSD changes. We've pretty much given up on merging macOS, and will plan to arrange our repo and workflow to be permanently down-stream, and have own copies of files for easier merging.

hey @lundman what the, why are you finally giving up?

andrewc12 · 2024-01-18T07:20:29Z

I'm trying to get smaller PRs merged:

lundman mentioned this pull request May 24, 2021

WIP: Add macOS support #11273

Closed

12 tasks

lundman force-pushed the macOS branch 2 times, most recently from 5e2d62d to 65a74e3 Compare May 24, 2021 07:57

ahrens assigned mmaybee May 24, 2021

behlendorf added the Status: Work in Progress Not yet ready for general review label May 24, 2021

rincebrain mentioned this pull request May 25, 2021

Actually enable CONFIG_ZFS in builds. openzfs/zfs-buildbot#228

Merged

lundman force-pushed the macOS branch 2 times, most recently from 4277da2 to 162912a Compare May 31, 2021 11:48

lundman force-pushed the macOS branch 6 times, most recently from ef3b4c8 to e7b6e73 Compare June 8, 2021 11:26

ilovezfs force-pushed the macOS branch from f90ceab to ef7f2a1 Compare June 9, 2021 06:16

lundman force-pushed the macOS branch 3 times, most recently from c7316f5 to ade3311 Compare June 10, 2021 01:01

lundman force-pushed the macOS branch 12 times, most recently from 84848e9 to 96f9ab0 Compare June 23, 2021 00:01

lundman force-pushed the macOS branch from 7780bfd to 184b03b Compare August 12, 2023 11:46

lundman force-pushed the macOS branch from 184b03b to b474bca Compare August 25, 2023 01:04

lundman force-pushed the macOS branch 5 times, most recently from 0d075ae to bc3f497 Compare October 20, 2023 05:10

lundman force-pushed the macOS branch from bc3f497 to d3c8762 Compare October 22, 2023 01:41

mcmilk suggested changes Oct 22, 2023

View reviewed changes

lundman force-pushed the macOS branch from d3c8762 to d68ad3b Compare October 23, 2023 00:11

lundman added 6 commits October 27, 2023 16:25

1: all macOS files

b317f05

part1: module/os/macos/* + include/os/macos/* This does then inlcude lib/*macos* as well, using grep so there are outliers, like manpage. Signed-off-by: Jorgen Lundman <[email protected]>

2: module/ changes

bf89419

part2: module/* Signed-off-by: Jorgen Lundman <[email protected]>

3: cmd/* and tests/*

12610e7

part3: cmd/* + tests/* Signed-off-by: Jorgen Lundman <[email protected]>

4: config/* scripts/* contrib/*

5b2f117

part4: config/, scripts/, contrib/* Signed-off-by: Jorgen Lundman <[email protected]>

5: include/* lib/* and top build

bb8e0a5

Bonus part5, the non-macos changes to include lib and the top level Makefiles Signed-off-by: Jorgen Lundman <[email protected]>

6: etc/*

57a596a

Signed-off-by: Jorgen Lundman <[email protected]>

lundman force-pushed the macOS branch from 2c888e2 to 57a596a Compare October 27, 2023 07:25

yurikoles reviewed Nov 5, 2023

View reviewed changes

andrewc12 and others added 3 commits November 14, 2023 09:21

MacOS: Build with n+1 cpus (#22)

136118a

Signed-off-by: Andrew Innes <[email protected]>

macOS: use portable paths for brewed deps (#23)

ab6764d

Signed-off-by: Yurii Kolesnykov <[email protected]>

Search for C++ and Obj C compilers (#21)

d9e3c90

Signed-off-by: Andrew Innes <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add macOS support to OpenZFS #12110

Add macOS support to OpenZFS #12110

lundman commented May 24, 2021 •

edited

Loading

lundman commented Jul 19, 2023

andrewc12 commented Oct 20, 2023 •

edited

Loading

mcmilk commented Oct 22, 2023 •

edited

Loading

mcmilk left a comment •

edited

Loading

lundman commented Oct 23, 2023

andrewc12 commented Oct 24, 2023 •

edited

Loading

andrewc12 commented Oct 24, 2023 •

edited

Loading

lundman commented Oct 24, 2023

yurikoles Nov 5, 2023 •

edited

Loading

andrewc12 Nov 6, 2023 •

edited

Loading

yurikoles Nov 6, 2023 •

edited

Loading

andrewc12 Nov 13, 2023 •

edited

Loading

andrewc12 Nov 14, 2023

andrewc12 commented Jan 17, 2024

andrewc12 commented Jan 18, 2024

Add macOS support to OpenZFS #12110

Are you sure you want to change the base?

Add macOS support to OpenZFS #12110

Conversation

lundman commented May 24, 2021 • edited Loading

Motivation and Context

Description

How Has This Been Tested?

Types of changes

Checklist:

lundman commented Jul 19, 2023

andrewc12 commented Oct 20, 2023 • edited Loading

mcmilk commented Oct 22, 2023 • edited Loading

mcmilk left a comment • edited Loading

Choose a reason for hiding this comment

lundman commented Oct 23, 2023

andrewc12 commented Oct 24, 2023 • edited Loading

andrewc12 commented Oct 24, 2023 • edited Loading

lundman commented Oct 24, 2023

yurikoles Nov 5, 2023 • edited Loading

Choose a reason for hiding this comment

andrewc12 Nov 6, 2023 • edited Loading

Choose a reason for hiding this comment

yurikoles Nov 6, 2023 • edited Loading

Choose a reason for hiding this comment

andrewc12 Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

andrewc12 Nov 14, 2023

Choose a reason for hiding this comment

andrewc12 commented Jan 17, 2024

andrewc12 commented Jan 18, 2024

lundman commented May 24, 2021 •

edited

Loading

andrewc12 commented Oct 20, 2023 •

edited

Loading

mcmilk commented Oct 22, 2023 •

edited

Loading

mcmilk left a comment •

edited

Loading

andrewc12 commented Oct 24, 2023 •

edited

Loading

andrewc12 commented Oct 24, 2023 •

edited

Loading

yurikoles Nov 5, 2023 •

edited

Loading

andrewc12 Nov 6, 2023 •

edited

Loading

yurikoles Nov 6, 2023 •

edited

Loading

andrewc12 Nov 13, 2023 •

edited

Loading