Homebrew formula #214

mhx · 2024-04-19T05:37:25Z

It would be helpful if the build instructions could be turned into a Homebrew formula.

alexmyczko · 2024-05-07T14:23:05Z

now that i got it building on debian (http://bananas.debian.net/debian/dwarfs/), it should be easy to create a homebrew formula with the help of https://github.com/mhx/dwarfs?tab=readme-ov-file#building-on-macos

99% tests passed, 9 tests failed out of 3095

Total Test time (real) =  27.37 sec

The following tests FAILED:
	2970 - dwarfs/tools_test.end_to_end/standalone (Subprocess aborted)
	2971 - dwarfs/tools_test.end_to_end/universal-tool (Subprocess aborted)
	2972 - dwarfs/tools_test.end_to_end/universal-symlink (Subprocess aborted)
	2973 - dwarfs/tools_test.mutating_and_error_ops/standalone (Subprocess aborted)
	2974 - dwarfs/tools_test.mutating_and_error_ops/universal-tool (Subprocess aborted)
	2975 - dwarfs/tools_test.mutating_and_error_ops/universal-symlink (Subprocess aborted)
	2976 - dwarfs/tools_test.categorize/standalone (Subprocess aborted)
	2977 - dwarfs/tools_test.categorize/universal-tool (Subprocess aborted)
	2978 - dwarfs/tools_test.categorize/universal-symlink (Subprocess aborted)
Errors while running CTest
Output from these tests are in: /Users/amyczko/homebrew/dwarfs/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
FAILED: CMakeFiles/test.util 
cd /Users/amyczko/homebrew/dwarfs/build && /opt/homebrew/Cellar/cmake/3.29.2/bin/ctest --force-new-ctest-process
ninja: build stopped: subcommand failed.

some tests failed, otherwise looks good, most difficult part will be the system extension with special reboot needed to install... any chance to skip the fuse part and use it without?

ok there are some binaries in build/ i guess those should be installed by the formula, and that'll be a good start...
when i have something, you'll find it at https://github.com/alexmyczko/homebrew-mac

binaries are quite large...

-rwxr-xr-x  1 amyczko  staff  3414184 May  7 18:37 dwarfs
-rwxr-xr-x  1 amyczko  staff  2437008 May  7 18:47 dwarfs.strip
-rwxr-xr-x  1 amyczko  staff   737296 May  7 18:47 dwarfs.strip.upx

mhx · 2024-05-07T19:19:51Z

Many thanks for looking into this!

some tests failed, otherwise looks good

These tests likely fail because macFUSE isn't fully enabled yet. It's exactly the same failures as when running in a docker container without proper privileges.

most difficult part will be the system extension with special reboot needed to install...

That should be taken care of by the macFUSE install. I agree that it's a real pain, but I don't think there's much that we can do here. (Happy to be convinced otherwise, though!)

any chance to skip the fuse part and use it without?

Yeah, I should have made it easier to disable these tests at run-time. Noted for the next release.

In the meantime, the simplest fix would be to just patch CMakeLists.txt and remove the tools_test line.

ok there are some binaries in build/ i guess those should be installed by the formula

The install target should take care of everything that's worth installing, which is essentially all binaries + all manpages.

binaries are quite large...

Do you think so? I'd consider them quite moderate at a few MiBs. :)

Anyway, really appreciate that you're looking into this!

mhx · 2024-05-07T19:41:48Z

any chance to skip the fuse part and use it without?

A much better way would be something like 2cb5542.

alexmyczko · 2024-05-08T07:42:28Z

As starting point I've created this tarball using git2deb: http://bananas.debian.net/macos/dwarfs/

alexmyczko · 2024-05-08T12:12:05Z

so download dwarfs.rb, run brew install ./dwarfs.rb and it fails:

https://github.com/alexmyczko/homebrew-mac/blob/main/dwarfs.rb

ah my tarball is bad, i'll get that fixed and soon there'll be a formula...

now it's:

-- Checking for module 'flac++>=1.4.2'
--   Found flac++, version 1.4.3
-- Checking for module 'libzstd>=1.5.2'
--   Found libzstd, version 1.5.6
-- Checking for module 'libxxhash>=0.8.1'
--   Found libxxhash, version 0.8.2
-- Checking for module 'fuse>=2.9.9'
--   No package 'fuse' found
-- Checking for module 'fuse3>=3.10.5'
--   No package 'fuse3' found
CMake Error at CMakeLists.txt:354 (message):
  No FUSE or FUSE3 library found


-- Configuring incomplete, errors occurred!

will need to check ccmake options if i can do something about it...

mhx · 2024-05-08T14:10:46Z

now it's:

-- Checking for module 'flac++>=1.4.2'
--   Found flac++, version 1.4.3
-- Checking for module 'libzstd>=1.5.2'
--   Found libzstd, version 1.5.6
-- Checking for module 'libxxhash>=0.8.1'
--   Found libxxhash, version 0.8.2
-- Checking for module 'fuse>=2.9.9'
--   No package 'fuse' found
-- Checking for module 'fuse3>=3.10.5'
--   No package 'fuse3' found
CMake Error at CMakeLists.txt:354 (message):
  No FUSE or FUSE3 library found


-- Configuring incomplete, errors occurred!

I'd guess that it's because the macfuse dependency is missing from the build script?

mhx · 2024-05-08T14:12:08Z

I think we should still depend on macfuse, but just not require that it is already fully set up (with reboots and stuff) when running the tests.

alexmyczko · 2024-05-08T14:21:56Z

and the brew formula is just called macfuse? i will try...

mhx · 2024-05-08T14:54:23Z

and the brew formula is just called macfuse? i will try...

Yeah, it's the 4th item in the brew install command from the instructions. :)

alexmyczko · 2024-05-08T18:05:54Z

hmmm

Cask 'dwarfs' is unreadable: wrong constant name #<Class:0x000000010fcb6af0>
Warning: Treating ./dwarfs.rb as a formula.
Warning: No available formula with the name "macfuse" (dependency of dwarfs). Did you mean afuse?
==> Searching for similarly named formulae and casks...
==> Casks
macfuse ✔                                                                                          macfuse@dev

To install macfuse ✔, run:
  brew install --cask macfuse ✔

mhx · 2024-05-08T19:07:45Z

Meh: https://github.com/orgs/Homebrew/discussions/3163#discussioncomment-2526187

mhx · 2024-05-08T19:25:43Z

Although this seems to suggest that a formula can depend on a cask...

mhx · 2024-05-08T19:25:50Z

Also interesting: Homebrew/homebrew-core#64491

mhx · 2024-05-08T19:26:55Z

Oh, here we go: https://github.com/gromgit/homebrew-fuse

mhx · 2024-05-08T19:28:51Z

This might help: https://github.com/gromgit/homebrew-fuse/blob/main/require/macfuse.rb
And this: https://github.com/gromgit/homebrew-fuse/blob/main/Formula/squashfuse-mac.rb

alexmyczko · 2024-05-09T08:50:32Z

now it's:
-- Checking for module 'flac++>=1.4.2'

-- Found flac++, version 1.4.3

-- Checking for module 'libzstd>=1.5.2'

-- Found libzstd, version 1.5.6

-- Checking for module 'libxxhash>=0.8.1'

-- Found libxxhash, version 0.8.2

-- Checking for module 'fuse>=2.9.9'

-- No package 'fuse' found

-- Checking for module 'fuse3>=3.10.5'

-- No package 'fuse3' found

CMake Error at CMakeLists.txt:354 (message):

No FUSE or FUSE3 library found

-- Configuring incomplete, errors occurred!
I'd guess that it's because the macfuse dependency is missing from the build script?

i tried with, without luck

alexmyczko · 2024-05-09T13:13:05Z

yay! i think the last two comments helped. i've got it soon

==> Fetching dwarfs
==> Downloading http://bananas.debian.net/macos/dwarfs/dwarfs-0.9.9.orig.tar.xz
Already downloaded: /Users/amyczko/Library/Caches/Homebrew/downloads/3a4ad90117afea0b5c8344658ca4a4984055b55d0b46dbe267796f2f4b59d8e3--dwarfs-0.9.9.orig.tar.xz
==> cmake .
==> make
Error: Empty installation

it's installing for me the binaries and manpages. do you need any libs installed or so?
got a better test suggestion?

dwarfs (v0.9.9-30-g2cb5542a5d on branch main [2024-05-07], fuse version 29)
built for arm64, Darwin-23.4.0, AppleClang 15.0.0.15000309

mhx · 2024-05-09T14:19:43Z

yay! i think the last two comments helped. i've got it soon

That is awesome!

==> Fetching dwarfs
==> Downloading http://bananas.debian.net/macos/dwarfs/dwarfs-0.9.9.orig.tar.xz
Already downloaded: /Users/amyczko/Library/Caches/Homebrew/downloads/3a4ad90117afea0b5c8344658ca4a4984055b55d0b46dbe267796f2f4b59d8e3--dwarfs-0.9.9.orig.tar.xz
==> cmake .
==> make
Error: Empty installation

it's installing for me the binaries and manpages. do you need any libs installed or so?

Binaries and manpages should be sufficient.

got a better test suggestion?

If you could get the following sequence of commands to run via homebrew that would be ideal (for the ninja version you need ninja as a dependency):

cmake -DWITH_TESTS=ON -GNinja, or at least cmake -DWITH_TESTS=ON
ninja or make -j$(sysctl -n hw.physicalcpu)
ninja test / make test, preferably with CTEST_PARALLEL_LEVEL=$(sysctl -n hw.physicalcpu) and DWARFS_SKIP_FUSE_TESTS=1 set in the environment
ninja install / make install (this should install everything needed)

dwarfs (v0.9.9-30-g2cb5542a5d on branch main [2024-05-07], fuse version 29)
built for arm64, Darwin-23.4.0, AppleClang 15.0.0.15000309

Once this is working I'll roll a new release so you can reference a release tag rather than a random commit.

alexmyczko · 2024-05-10T06:04:53Z

I guess we have to build and be installable without macfuse. Homebrew/homebrew-core#171264 (comment)

mhx · 2024-05-10T08:08:36Z

I guess we have to build and be installable without macfuse. Homebrew/homebrew-core#171264 (comment)

That kinda defeats the whole point of DwarFS and is going to be confusing and annoying at the same time. Confusing because likely people installing it won't know they can't use the FUSE part until they've finished installation. And annoying because then they have to re-install the whole thing (which takes a while to build) after installing macFUSE.

My personal (likely biased) opinion is that banning the dependency on macFUSE is... suboptimal.

There are a couple of ways that I can see this proceeding:

As you suggest, make DwarFS build/install without FUSE. Then it'd depend on whether or not macFUSE is already installed if you get the FUSE driver or not.
Add the formula to gromgit/homebrew-fuse rather than homebrew-core.
Have a dwarfs formula in homebrew-core, with a :linux dependency, and a dwarfs-mac formula in homebrew-fuse.
Have libdwarfs, dwarfs-tools, dwarfs (:linux) formulae, all in homebrew/core, and a dwarfs-mac formula in homebrew/fuse. The latter 3 formulae would all have a dependency on libdwarfs; dwarfs-tools would be installable without the dependency on macFUSE.

I personally like option (4) the most, but it's going to be a significant amount of work on the codebase. Until we get there, (2)/(3) are probably my preference.

alexmyczko · 2024-05-11T06:11:01Z

for homebrew-core
it is already there https://github.com/alexmyczko/homebrew-mac
exercise for someone else, i will do the debian package, that devuan, raspbian, ubuntu will copy
no idea

mhx · 2024-05-18T16:15:39Z

for homebrew-core

it is already there https://github.com/alexmyczko/homebrew-mac

exercise for someone else, i will do the debian package, that devuan, raspbian, ubuntu will copy

no idea

I'm trying to get the code into a state where (4) is possible.

mhx · 2024-08-14T19:57:04Z

Hey, it's been a while!

It took only slightly more than 300 commits to pull this off. :)

Please, if you have time, could you try out dwarfs-0.10.0? This release allows modular builds, i.e. you can in theory provide formulae for libdwarfs, dwarfs-tools and dwarfs-fuse, just as mentioned before.

The release should build fine on macOS, both X64 and ARM64 (I've added these to the CI).

It should be fairly straightforward to build the different modules on macOS:

$ tar xf dwarfs-0.10.0.tar.xz

$ cmake -B build-lib -S dwarfs-0.10.0 -GNinja -DWITH_LIBDWARFS=ON -DWITH_TOOLS=OFF -DWITH_FUSE_DRIVER=OFF
$ cmake --build build-lib
$ sudo cmake --install build-lib

$ cmake -B build-tools -S dwarfs-0.10.0 -GNinja -DWITH_LIBDWARFS=OFF -DWITH_TOOLS=ON -DWITH_FUSE_DRIVER=OFF
$ cmake --build build-tools
$ sudo cmake --install build-tools

$ cmake -B build-fuse -S dwarfs-0.10.0 -GNinja -DWITH_LIBDWARFS=OFF -DWITH_TOOLS=OFF -DWITH_FUSE_DRIVER=ON
$ cmake --build build-fuse
$ sudo cmake --install build-fuse

Of course you can also still build everything, enable tests, etc.

Let me know if you need any help!

mhx · 2024-08-15T13:18:21Z

With some help from our AI overlords, I came up with two formulae.

`libdwarfs.rb`

class Libdwarfs < Formula
  desc "A fast high compression read-only file system for Linux, Windows, and macOS"
  homepage "https://github.com/mhx/dwarfs"
  url "https://github.com/mhx/dwarfs/releases/download/v0.10.0/dwarfs-0.10.0.tar.xz"
  sha256 "c01ae59d4662e4f027a7c8a5934b7aebe6edc4f7affd836fa73e6d861d18bf35"
  license "GPL-3"

  depends_on "cmake" => :build
  depends_on "ninja" => :build
  depends_on "googletest" => :build
  depends_on "pkg-config" => :build
  depends_on "libarchive"
  depends_on "brotli"
  depends_on "lz4"
  depends_on "xz"
  depends_on "zstd"
  depends_on "gflags"
  depends_on "glog"
  depends_on "parallel-hashmap"
  depends_on "boost"
  depends_on "double-conversion"
  depends_on "libevent"
  depends_on "fmt"
  depends_on "utf8cpp"
  depends_on "xxhash"
  depends_on "flac"

  patch do
    url "https://github.com/mhx/dwarfs/releases/download/v0.10.0/dwarfs-system-gtest.patch"
    sha256 "573656e33a171017046a32500b9ebe50ebd117df16212e6ef8f8d4b034618210"
  end

  def install
    args = std_cmake_args + %W[
      -DWITH_TESTS=ON
      -DFOLLY_USE_JEMALLOC=OFF
      -DWITH_MAN_PAGES=ON
      -DWITH_LIBDWARFS=ON
      -DWITH_TOOLS=OFF
      -DWITH_FUSE_DRIVER=OFF
      -DENABLE_PERFMON=ON
      -DTRY_ENABLE_FLAC=ON
      -DENABLE_RICEPP=ON
      -DENABLE_STACKTRACE=OFF
      -DPREFER_SYSTEM_GTEST=ON
    ]

    system "sed", "-i", "", "s|PARALLEL_HASHMAP_REQUIRED_VERSION 1.3.8|PARALLEL_HASHMAP_REQUIRED_VERSION 1.0.0|", "CMakeLists.txt"
    system "cmake", "--fresh", "-GNinja", "-S", ".", "-B", "build", *args
    system "cmake", "--build", "build", "--parallel"
    system "ctest", "--test-dir", "build", "--output-on-failure", "--parallel", ENV.make_jobs
    system "cmake", "--install", "build"
  end
end

`dwarfs-tools.rb`

class DwarfsTools < Formula
  desc "A fast high compression read-only file system for Linux, Windows, and macOS"
  homepage "https://github.com/mhx/dwarfs"
  url "https://github.com/mhx/dwarfs/releases/download/v0.10.0/dwarfs-0.10.0.tar.xz"
  sha256 "c01ae59d4662e4f027a7c8a5934b7aebe6edc4f7affd836fa73e6d861d18bf35"
  license "GPL-3"

  depends_on "cmake" => :build
  depends_on "ninja" => :build
  depends_on "googletest" => :build
  depends_on "libdwarfs"

  def install
    args = std_cmake_args + %W[
      -DWITH_LIBDWARFS=OFF
      -DWITH_TOOLS=ON
      -DWITH_FUSE_DRIVER=OFF
      -DWITH_MAN_PAGES=ON
      -DWITH_MAN_OPTION=ON
      -DPREFER_SYSTEM_GTEST=ON
    ]

    system "cmake", "--fresh", "-GNinja", "-S", ".", "-B", "build", *args
    system "cmake", "--build", "build", "--parallel"
    system "ctest", "--test-dir", "build", "--output-on-failure", "--parallel", ENV.make_jobs
    system "cmake", "--install", "build"
  end
end

These work fine, as far as I can tell.

The formula for the macFUSE driver should be very similar to dwarfs-tools, but I can't figure out how to model the macFUSE dependency properly. Feel free to use these as a starting point.

alexmyczko · 2024-08-16T11:38:34Z

what about you submit them to brew?

mhx · 2024-08-16T17:12:31Z

what about you submit them to brew?

Will give it a try!

mhx · 2024-08-25T10:39:11Z

Homebrew/homebrew-core#181569 has been merged.

Now it's just a formula for the FUSE driver that's missing.

mhx · 2024-10-19T15:50:35Z

PR for the FUSE driver formula: gromgit/homebrew-fuse#60

mhx added help wanted Extra attention is needed good first issue Good for newcomers labels May 4, 2024

mhx added a commit that referenced this issue May 7, 2024

chore: allow skipping of FUSE tests (should help with gh #214)

2cb5542

alexmyczko mentioned this issue May 10, 2024

dwarfs 0.9.9 (new formula) Homebrew/homebrew-core#171264

Closed

6 tasks

mhx added fixready and removed help wanted Extra attention is needed good first issue Good for newcomers labels Oct 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Homebrew formula #214

Homebrew formula #214

mhx commented Apr 19, 2024

alexmyczko commented May 7, 2024 •

edited

Loading

mhx commented May 7, 2024

mhx commented May 7, 2024

alexmyczko commented May 8, 2024

alexmyczko commented May 8, 2024 •

edited

Loading

mhx commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 9, 2024

alexmyczko commented May 9, 2024 •

edited

Loading

mhx commented May 9, 2024

alexmyczko commented May 10, 2024

mhx commented May 10, 2024

alexmyczko commented May 11, 2024 •

edited

Loading

mhx commented May 18, 2024

mhx commented Aug 14, 2024

mhx commented Aug 15, 2024 •

edited

Loading

alexmyczko commented Aug 16, 2024

mhx commented Aug 16, 2024

mhx commented Aug 25, 2024

mhx commented Oct 19, 2024

Homebrew formula #214

Homebrew formula #214

Comments

mhx commented Apr 19, 2024

alexmyczko commented May 7, 2024 • edited Loading

mhx commented May 7, 2024

mhx commented May 7, 2024

alexmyczko commented May 8, 2024

alexmyczko commented May 8, 2024 • edited Loading

mhx commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

mhx commented May 8, 2024

alexmyczko commented May 9, 2024

alexmyczko commented May 9, 2024 • edited Loading

mhx commented May 9, 2024

alexmyczko commented May 10, 2024

mhx commented May 10, 2024

alexmyczko commented May 11, 2024 • edited Loading

mhx commented May 18, 2024

mhx commented Aug 14, 2024

mhx commented Aug 15, 2024 • edited Loading

libdwarfs.rb

dwarfs-tools.rb

alexmyczko commented Aug 16, 2024

mhx commented Aug 16, 2024

mhx commented Aug 25, 2024

mhx commented Oct 19, 2024

alexmyczko commented May 7, 2024 •

edited

Loading

alexmyczko commented May 8, 2024 •

edited

Loading

alexmyczko commented May 9, 2024 •

edited

Loading

alexmyczko commented May 11, 2024 •

edited

Loading

mhx commented Aug 15, 2024 •

edited

Loading

`libdwarfs.rb`

`dwarfs-tools.rb`