Distributing quarks with binary components

[capocasa]

No description provided.

[capocasa]

So while I intended to mainly discuss implementation details for a source-based distribution of Quarks containing plugins that require building, feedback from sc-dev said that binary distributing would be preferable.

If we go the binary route, I agree that github releases are a great way to do it.

Perhaps my preference for automatically building from source comes from working with linux- installing a build system that works is just a one-liner in the shell and is very standardized and reliable, while having to setup something like homebrew on OSX just to get quarks would suck. I'll check out how the ruby folks did it for something like nokogiri, an Ruby SSL package that compiles. In any case, another thing to keep in mind is that linux has many, many different combinations of processor architecture, word length and executable format. Perhaps the pragmatic choice is to ship binaries for OSX and Windows, and allow linux users to autobuild. There is nothing more frustrating as to be an arch user and you have to do a manual build from source because the packages are only for ubuntu and redhat. Further, source build would also support the likes of FreeBSD, raspberry pi and the Bela.

Now I've got pretty good sclang knowledge so I can expand the Quark system in whatever way we decide, but I don't really have the skills to set up and maintain a CI system for the quarks, however that does seem to be the most quark-developer friendly option. I suspect that if we have quark developers build on their own box, getting cross-platform support would require every developer setting up a transpiling toolchain to OSX, Windows and Linux, which seems unreasonable to me, so we would likely get a lot of single-OS quarks. I think we should go to considerable lengths to avoid that.

[scztt]

There are some basic requirements that the quarks system needs in order to make something like this feasible. Here's the list of things off the top of my head, but there may be more - these should be incorporated somehow, as I don't think a binary system will work without them?

1. Proper SuperCollider / sclang version locking
   Quarks need to be able to specify theeir sclang version requirements in a standard way. Major upgrades to the quarks system like this will require a specific sclang version, and the quark file needs to be able to express this. There is currently a function that an returns a bool representing sclang support, but this requires each quark (correctly) implement version comparison - and, it's infeasible (and currently impossible) to run this on every released version of a quark to determine support.

2. Deterministic quark version dependencies
   Quarks need to be able to specify dependency constraints (e.g. mathlib >= 1.5) and properly resolve them. Currently, if a quark depends on another quark of version X, it is not guaranteed to actually get version X, depending on e.g. install order, order git operations, etc. This is already bad for sclang quarks, but this will be extra bad for source-compiled or binary quarks, where a different version could mean compile failures or runtime crashes.

3. A way to specify downloadable (e.g. non Git based) dependencies
   Quarks need a way to specify binary dependencies (things like pre-built binaries or resource files) and download them to a reasonable location when the Quark is checked out. sclang can already manage downloading, but there will be some additional requirements to place downloaded files with the rest of the Quark files (they shouldn't in the Quark folder itself because it's a git repository...). Probably, this means checking out the git part of the quark, downloading the binary part, and then asseembling everything into to a third location.

4. Specifying OS version requirements
   It's unlikely that most quarks will support all N possible operating systems - Quarks will need a straightforward way to indicate OS support, to prevent users from trying to install them in an unsupported scenario. At the very least, this would need to be e.g. major OS version number?

This is all I can think of at the moment - anyone else have thoughts here?

[scztt]

One possible strategy to ease the engineering load here... Basically platforms have at least one package manager that can already pull dependencies, build them or install binaries, manage versions etc... What if the binary distribution part of this was simply handled by the proper OS package manager, plus a Quarks PPA / tap that we maintain? This would mean quark owners would only need to specify e.g. a brew formula to get their quark working - for things like server plugins, this would be total boilerplate and easily copy-paste-able from a template. This feels easier than making quark maintainers write separate build scripts for every platform, and more reliable because the package manager can take care of a lot of the heavy lifting of e.g. build toolchains, other dependencies, etc.

Then, a quark file could have something like:

externalDependencies: (
    mac: (
        brew: (
            fftw: "1.0",
        )
    ),
    windows: (
        choco: (
           fftw: "1.0"
        )
    )
)

[Sciss]

I think it was already discussed on sc-dev with Christof (I haven't really looked into it), but it would be worth mentioning that one should probably look into the externals management of Pure Data ("Deken").

[capocasa]

look into the externals management of Pure Data ("Deken").

Done.

The Deken downloader is a TCL script that does approximately what Quark.gui does.

The Deken packager is made up of a shell script, a python script, and a script in hy, a lisp-syntax for python.

The shell and python scripts do some argument wrangling while the hy script does the heavy lifting, which consists of scanning a compiled library and deriving architecture, OS version, and executable format, and then packing it into a zip file with a name to that extent, e.g. osmething like myExternal-x86_64-darwin4-macho.zip or myExternal-armv6-linux-ELI5.zip, and uploads it to their package repository. That's it. The actual build process is intended to be done on each target machine individually using a boilerplate Makefile.

[capocasa]

My personal role model for quarks are Ruby Gems, which are like Quarks for ruby, because they pioneered the package script idea, they are a mature system with excellent usability and code quality, and because sclang has a tradition of borrowing from Ruby since the very beginning.

So I investigated Nokogiri as a stand-in for all binary Gems, which is one of the most popular ruby packages, which depends on compiling a ruby binary extension that depends on libxml and libxslt, two C-language XML libraries. This is a good role model because it is so widely used, and because it distributes a binary library that also depends on an existing C library, so it covers all use cases.

So it turns out that Nokogiri agrees with Christof, Brian and Luke for Windows insofar as it distributes a binary library and does not attempt to perform a build on the target system by default, but that is still available as an advanced option. However, for OSX, Nokogiri depends on installing the XCode command line tools, an install of which can be triggered with a single command. For various flavors of Linux, a package called build-essential or similar can also be installed with a single command, or via familiar GUI.

So I would like to use this to reiterate that building on target systems does not have to mean difficulty for the user at all, at least on unix-based systems such as OSX and Linux. Further, helpful error messages or even help installing dependencies could be displayed with a simple command-line availability check, just like it is done with git.

The benefits are quite simply that it is much, much easier to maintain a binary repository for a single system instead of three, and that full cross-platform compatibility can be achieved this way with the least possible amount of effort, resulting in a much richer ecosystem across platforms.

[capocasa]

One possible strategy to ease the engineering load here... Basically platforms have at least one package manager that can already pull dependencies, build them or install binaries, manage versions etc...

That's a cool idea to leverage existing systems!

My experience attempting to install python packages from the linux package manager together with packages installed by pip, a python package manager, indicates that what you are proposing is actually a much more difficult technical problem than extending Quarks to support binary, because three different systems and their packages must be integrated with Quarks. I would therefore prefer to require packages to supply their own dependencies as source, perhaps optionally linking to whatever system packages are present.

Proper SuperCollider / sclang version locking

My prototype Quark.build already reads the version number from sclang to download the right SC headers for bulding, what I had in mind is that a switch is detected and a notification to run Quarks.updateAll (or something) is run. Thinking about it more, we should probably just put all binary quarks in version specific directories so you can use the same install based on various SC versions with no conflict.

Deterministic quark version dependencies
Oh, that doesn't exist yet? Yeah I can implement that.

A way to specify downloadable (e.g. non Git based) dependencies
I suppose we could use zip or tar.gz files for that, like the Arch linux package install script does. I'd consider it an edge case though, not something I'd include in version 0.0.1

Specifying OS version requirements

Same same- that doesn't exist yet? Weird.

[mossheim]

We discussed this on the developer call today. A couple topics came up:

• overall everyone would really like to see this happen, it's just a matter of doing it in a way that doesn't create more problems down the road
• this solution should place a strong emphasis on ease of use for non-developers, plugin authors, and project maintainers. in particular:
  • root permissions should not be required to install plugins, since this is often an impossibility at universities and other institutions. this probably limits our options on Windows and macOS
  • end users should not be required to download or install extra tools themselves if possible
  • if problems occur, end users should be given an easy way to report that and/or fix it
  • plugin authors should have a way to confirm that any scripts they write will work (CI)
• something to keep in mind is that SC is relatively small and its audience is not just other developers, and so solutions that work for general-purpose programming languages may not work here. in particular, users are less likely to contribute back to maintenance of the package ecosystem, as many cannot diagnose or resolve issues they encounter on their own
• for Linuxes, from-source builds is probably still a necessity given the number of flavors out there.
• would be worthwhile to look into using the trade-offs of using Homebrew or some other existing solution to distribute these

please edit this comment if there's anything i forgot to add!

[nhthn]

agree with brian, i think this is a really important direction to go in. we need to think bigger about the large-scale implications. some rhetorical questions for us to consider:

• what are the most pressing issues facing users and developers in the distribution of binary artifacts for SC?
• what should the process look like for a user installing a package?
• what should the process look like for a developer creating a package?
• what are our constraints? (users may not be able to do X, would not be reasonable to ask developers to do Y, etc.)
• what is the impact on non-sclang clients?

[capocasa]

Thanks for the great summary, Brian, and thanks again to everyone for the cool discussion on the call.

• Agree strongly with the ease of use requirement, and if that can't be done with source, binary it is. This would apply to OSX now that we know total userland is a requirement.
• Agree on full automation of all installation steps
• Agree on easy reporting but problem underdefined- is scynth.org easy or does it have to be in the GUI?
• Agree on CI, for both developers and user-security
• I do think learning form major languages is a good idea where requirements overlap.
• reliable binary on linux would require about 20 or so different builds, I am guessing, but we could do x64_86-ELF and leave the rest to source.
• I've never heard of system specific package manager like homebrew that live in userland

Also agree with Nathan's input! Let me just put out what I've got so far on your questions:

• Users mainly don't want to know or care that binary artifacts exist, while developers need to minimize sources of friction
• The package installation process should look exactly the same as it does for installing packages without binary artifacts
• Developers clone a boilerplate repository and need to customize it only for advanced usage. Build system installation is well documented and straight forward to install even for inexperienced developers. Dependencies should be included in the repository as code or submodules, linking to system-installed libraries should be discouraged. Considering languages other than C++ should be deferred.
• Constraints:
  • Developers should not need to build binaries for distribution themselves for security reasons
  • Installation should be a single click or command
  • The scope of the RFC is binary artifacts only- easy of use must match that of existing quarks. Then, usability of both should be improved in seperate RFCs. Changes should be minimal to achieve that goal.
  • Confirm that binary libraries can be downloaded and used at universities as long as they live in userland
• non-sclang clients need to either use sclang to install quarks, or install quarks themselves. This is probably easier than it sounds because providing the binary repository is the hard part. Once it's in place, downloading the binaries in addition to the source would appear to be managaeble.

[capocasa]

I updated the RFC to reflect the state of discussion after the developer call to reflect the requirement of non-root user installation, and hence of binary distribution. Please comment as you see fit!

[joshpar]

@capocasa - do you feel like this is now at a stage for consideration to work on for 3.12?

[capocasa]

@joshpar Yeah, I think it is!