Discussion regarding creating an open macOS SDK package.

[mingwandroid]

I think no one wants to tell people to use old macOS's or download stuff from people with hackery sounding usernames on github, at least I do not.

Now that conda-forge/staged-recipes#9557 has happened, we should be able to allow end-users to use their system SDKs via CONDA_BUILD_SYSROOT, but that's not ideal for ABI reasons.

But I think we should provide our own SDKs. The official SDKs are an amalgamation of various mostly BSD licensed projects (mach-o kernel, dyld, a few more). I think we can now generate .tbd files from Apple's .dylibs and we can also 'whiteroom' that if people are concerned - you literally get one person to describe, via voice, the functions and their prototypes! - but personally I'm not really very concerned. So I think, given a simple bash script we could make something we can use and recommend to our users.

cc @conda-forge/core, @chrisburr (you're in core anyway I guess, but just in case), @katietz, @msarahan.

[isuruf]

We can ask NumFOCUS legal if we can generate the .tbd files from tapi and distribute those.
cc @scopatz

What about modulemap files?

For the header files, we need to go through 8000 files.

[mingwandroid]

Maybe need some tooling to determine licenses and automate it, perhaps Python is better than bash if we need to do serious engineering?

[mingwandroid]

What about modulemap files?

I'm not familiar with those, possibly that knowledge has left my brain!

[mingwandroid]

We should also think about XQuartz development packages unless conda-forge has detached from all XQuartz dependencies entirely? If we package XQuartz we could make it depend on an SDK package and use a symlink to impose it into those SDK hierarchies.

[ocefpaf]

We should also think about XQuartz

We have at least one package that installs XQuartz via Homebrew. It would be nice to have that as part of the SDK.

[jakirkham]

IIRC the issue with using .tbd was Apple hadn’t upstreamed their support for TAPI into LLVM/Clang. Is that not still an issue? If not, agree this would be a great solution!

[mingwandroid]

All is good now. They have upstreamed it yes and Isuru has taken the mac assembler and linker over the line now and updated them to the latest we can get our hands on.

[scopatz]

I am not familiar with this legal issues here. Could someone catch me up?

[isuruf]

SDK is officially not redistributable, but we are trying to workaround that.
There are several types of files.

1. .h, .defs, .r, .x - Header files. these are mainly under open source licenses.
2. .tbd - these are ABI definitions for shared libraries. Here's an example, https://stackoverflow.com/questions/31450690/why-xcode-7-shows-tbd-instead-of-dylib
   These are generated from .dylib files and to avoid license issues, we can regenerate them ourselves. (These are APIs and are probably not copyrightable. Disclaimer: IANAL)
3. .modulemap, .map - these are like .tbds, but for header files. Not sure we can generate them. We can ignore them and not redistribute as they are used by Obj-C atm.
4. .o files - files like crt1.o. These are compiled into executables and dylibs, so they should be redistributable. (otherwise we can't distribute any macosx binaries)
5. pkg-config files. These are all from open-source projects. (Should double check)
6. .plist files. Not sure what to do with these.
7. Others. .apinotes, .rb - can ignore and not redistribute
8. .a files. ignore and not redistribute?

(2.) was the main issue because we had .dylibs in older SDKs and they were definitely not redistributable. We didn't have a linker that could link .tbds, but we do now. So, we don't have to ship .dylibs and we can just ship .tbds.

[scopatz]

Wow what a mess.

1. .h, .defs, .r, .x - We need to be careful here and make sure each is actually open source, preferably sourcing them from upstream packages if possible.
2. .tbd - Let's regenerate
3. .modulemap, .map - Let's ignore if there for the moment.
4. .o - I actually doubt we can redistribute object files unless they are open source.
5. pkg-config - again we need to be careful and double check.
6. .plist - are they needed?
7. .apinotes, .rb - yep let's ignore
8. .a - we are trying to move away from static libraries anyway, so it is probably a good idea to ignore

[mingwandroid]

Sounds like a plan.

[isuruf]

@mingwandroid, what are your thoughts on 4 and 6 above?

[mingwandroid]

For 4. I expect we can build the crt*.o files, I remember seeing the source somewhere when looking at fully static exes on mac years ago. If not I believe we may want to look into interop rules around monopolies. Otherwise we could look at official clang releases or require those to come from an official sdk (unfortunate lingering dep on command-line-tools or Xcode?)

For 6. plists are kind of like manifest files. They instruct cocoa about setting up the execution env (to access things like HiDPI you need a plist, also mime/file associations are done that way). I doubt we need any from the SDK. We do use create and use them in python.app and RStudio for example.

[mingwandroid]

Pinging @katietz here since he started mingw-w64 which tackled very similar issues as this open macOS sdk is very close to mingw-w64 in purpose (once combined with our compilers it's basically the same deal and I guess Kai will have looked into legalities we are discussing).

[isuruf]

CSU package source is at https://opensource.apple.com/source/Csu/

[mingwandroid]

I messed about with this to generate patches with headers needed to compile cctools and what not:

https://github.com/AnacondaRecipes/aggregate/blob/master/cctools-ld64-feedstock/recipe/copy-macOS-sdk-headers

Very much an engineers script, but it has URLs and stuff.

[minrk]

I'm absolutely thrilled to see progress on this, and happy to help with testing, etc. if I can.

[isuruf]

There are some headers like https://github.com/phracker/MacOSX-SDKs/blob/master/MacOSX10.9.sdk/usr/include/cache.h without licenses. @scopatz, could you ask NumFOCUS legal whether files like these are re-distributable?

[isuruf]

@minrk, you can certainly help.

1. Download https://github.com/phracker/MacOSX-SDKs/releases/download/10.13/MacOSX10.9.sdk.tar.xz and extract
2. Install clangxx_osx-64=8.0.1 from conda-forge.
3. Run https://github.com/isuruf/OpenMacOSSDK/blob/master/create.py on the extracted folder.
4. Try compiling software with that SDK. (set CONDA_BUILD_SYSROOT to the new SDK and compile)

[scopatz]

@isuruf - where is the license for these files? I couldn't find it in the repo?