Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide darwin/arm64 builds of jq #2386

Closed
cfstras opened this issue Dec 16, 2021 · 19 comments · Fixed by #2665
Closed

Provide darwin/arm64 builds of jq #2386

cfstras opened this issue Dec 16, 2021 · 19 comments · Fixed by #2665

Comments

@cfstras
Copy link

cfstras commented Dec 16, 2021

Would it be possible to release a jq version with a built binary for M1 Macs?

I didn't find any release scripts for mac, otherwise I'd made a PR :)


Using the build instructions in the README, I was able to produce a arm64 macOS binary of jq. I haven't extensively tested it, but it seems to work fine.

However, I saw an issue with make check:

/Library/Developer/CommandLineTools/usr/bin/make  check-recursive
Making check in modules/oniguruma
Making check in src
Making check in test
/Library/Developer/CommandLineTools/usr/bin/make  testc testp testcu
make[4]: `testc' is up to date.
make[4]: `testp' is up to date.
make[4]: `testcu' is up to date.
/Library/Developer/CommandLineTools/usr/bin/make  check-TESTS
PASS: testc
PASS: testp
PASS: testcu
============================================================================
Testsuite summary for onig 6.1.3
============================================================================
# TOTAL: 3
# PASS:  3
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================
Making check in sample
/Library/Developer/CommandLineTools/usr/bin/make  encode listcap names posix simple sql syntax user_property bug_fix
make[4]: `encode' is up to date.
make[4]: `listcap' is up to date.
make[4]: `names' is up to date.
gcc -DHAVE_CONFIG_H -I. -I../src  -I../src -I/usr/local/include   -g -O2 -MT posix.o -MD -MP -MF .deps/posix.Tpo -c -o posix.o posix.c
posix.c:94:3: error: implicit declaration of function 'onig_end' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
  onig_end();
  ^
1 error generated.
make[4]: *** [posix.o] Error 1
make[3]: *** [check-am] Error 2
make[2]: *** [check-recursive] Error 1
make[1]: *** [check-recursive] Error 1
make: *** [check] Error 2

@jessesuen
Copy link

There's a macOS arm64 version installable via brew:

$ file `which jq`
/opt/homebrew/bin/jq: Mach-O 64-bit executable arm64

You can see the patch which was necessary to get this to work on macOS:

https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/jq.rb

@cfstras
Copy link
Author

cfstras commented Dec 22, 2021

Looks like that fix already has made it into master (the file I have generated locally from the build instructions already has it applied).

The homebrew bottle does not run make check currently...

@grnch
Copy link

grnch commented Dec 31, 2021

Could we still add an OSX ARM64 build to the Downloads page, for those of us who don't use Homebrew?

@deric4
Copy link

deric4 commented Jan 11, 2022

@cfstras @grnch I've been able to use the jq-osx-amd64 binary from the downloads page without any problems (so far 🤞 ) on my M1 with rosetta installed.

$ uname -m
arm64

$ system_profiler SPHardwareDataType
Hardware:

    Hardware Overview:

      Model Name: MacBook Pro
      Model Identifier: MacBookPro18,3
      Chip: Apple M1 Pro
      Total Number of Cores: 8 (6 performance and 2 efficiency)
      Memory: 16 GB
      System Firmware Version: 7429.30.65
      OS Loader Version: 7429.30.65
      ...

$ file jq-osx-amd64 
jq-osx-amd64: Mach-O 64-bit executable x86_64

$ chmod +x jq-osx-amd64

$ ./jq-osx-amd64 --version
jq-1.6

@grnch
Copy link

grnch commented Jan 12, 2022

@deric4 Of course you can run the Intel binary on an M1 Mac, but it will run under Rosetta 2 emulation which will impact its performance. You won't notice any difference on small JSON files, but if you use jq to scan large machine generated JSON dumps it would sure be nice to have a native M1 executable running at peak performance.

@deric4
Copy link

deric4 commented Jan 12, 2022

... You won't notice any difference on small JSON files, but if you use jq to scan large machine generated JSON dumps it would sure be nice to have a native M1 executable running at peak performance.

That makes sense! I typically am only dealing with, at most, a few MB. Just for fun (and to see if casual usage would be noticeably impacted) I tried a couple of the jeopardy benchmark tests mentioned in the wiki.
https://github.com/stedolan/jq/wiki/X---Experimental-Benchmarks

$ time md5 jeopardy.json 
MD5 (jeopardy.json) = 2075398fa049b1c00223b2279ca5281d

real	0m0.133s
user	0m0.124s
sys	0m0.019s
$ time ./jq-osx-amd64 length jeopardy.json 
216930

real	0m1.226s
user	0m1.113s
sys	0m0.061s

Note: the schema.jq gist referenced seems to have been updated since the wiki entry was created

Assuming this version of schema.jq was used originally : https://gist.github.com/pkoppstein/a5abb4ebef3b0f72a6ed/e2535a035c6f428b45d5b843d7e256c3d98ee069

$ time ./jq-osx-amd64 -f schema2017.jq jeopardy.json > jeopardy2017.schema.json

real	0m6.374s
user	0m6.259s
sys	0m0.066s

$ cat jeopardy2017.schema.json 
{
  "air_date": "string",
  "answer": "string",
  "category": "string",
  "question": "string",
  "round": "string",
  "show_number": "string",
  "value": "string"
}

current version of schema.jq

$ time ./jq-osx-amd64 -f schema.jq jeopardy.json > jeopardy.schema.json

real	0m15.806s
user	0m15.672s
sys	0m0.078s
$ cat jeopardy.schema.json 
{
  "air_date": "string",
  "answer": "string",
  "category": "string",
  "question": "string",
  "round": "string",
  "show_number": "string",
  "value": [
    "+",
    "null",
    "string"
  ]
}

I'll stop creating noise in the issue but thought it was pretty cool that it doesn't seem to be any worse than a machine from back in 2018 😅

@cfstras
Copy link
Author

cfstras commented Jan 12, 2022

Here's my manually compiled version (jq-1.6-145-ga9f97e9-dirty) versus the 1.6 release on amd.

➜ hyperfine --warmup 2 "./jq-arm length jeopardy.json"
Benchmark 1: ./jq-arm length jeopardy.json
  Time (mean ± σ):     492.8 ms ±   8.6 ms    [User: 450.7 ms, System: 38.1 ms]
  Range (min … max):   484.2 ms … 508.9 ms    10 runs

➜ hyperfine --warmup 2 "./jq-amd length jeopardy.json"
Benchmark 1: ./jq-amd length jeopardy.json
  Time (mean ± σ):      1.190 s ±  0.022 s    [User: 1.111 s, System: 0.057 s]
  Range (min … max):    1.169 s …  1.229 s    10 runs

With schema (the version you linked, after uncommenting the last line):

➜ hyperfine --warmup 2 "./jq-arm -f schema.jq jeopardy.json"
Benchmark 1: ./jq-arm -f schema.jq jeopardy.json
  Time (mean ± σ):      3.543 s ±  0.045 s    [User: 3.479 s, System: 0.046 s]
  Range (min … max):    3.482 s …  3.618 s    10 runs
➜ hyperfine --warmup 2 "./jq-amd -f schema.jq jeopardy.json"
Benchmark 1: ./jq-amd -f schema.jq jeopardy.json
  Time (mean ± σ):      6.550 s ±  0.102 s    [User: 6.454 s, System: 0.071 s]
  Range (min … max):    6.430 s …  6.740 s    10 runs

With no data at all, measuring startup time:

➜ hyperfine --warmup 2 "echo '{}' | ./jq-arm .test"
Benchmark 1: echo '{}' | ./jq-arm .test
  Time (mean ± σ):       3.1 ms ±   4.5 ms    [User: 1.3 ms, System: 0.9 ms]
  Range (min … max):     0.0 ms …  63.1 ms    249 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

➜ hyperfine --warmup 2 "echo '{}' | ./jq-amd .test"
Benchmark 1: echo '{}' | ./jq-amd .test
  Time (mean ± σ):      34.9 ms ±   2.1 ms    [User: 24.4 ms, System: 2.1 ms]
  Range (min … max):    31.5 ms …  45.6 ms    69 runs

So: I get ~2x performance by using the native ARM build over rosetta, and startup time goes down from 30ms to 1ms.
I'd say it's definitely worth it :)

@slamdev
Copy link

slamdev commented Mar 28, 2022

you can use arm 1.6 binary from https://github.com/slamdev/rules_jq/releases/download/0.0.1/jq-osx-arm64 until the official build is released.

hwchiu added a commit to hwchiu/arkade that referenced this issue Apr 3, 2022
- No official arm64 build for now but currnet amd64 binary works for the M1 env, should change to the arm64 binary once jq have official arm64 version in its Github release.
- Reference Github issue: jqlang/jq#2386

Signed-off-by: HungWei Chiu <[email protected]>
hwchiu added a commit to hwchiu/arkade that referenced this issue Apr 3, 2022
- No official arm64 build for now but the current amd64 binary works for the M1 env, should change to the arm64 binary once jq has official arm64 version in its Github release.
- Reference Github issue: jqlang/jq#2386

Signed-off-by: HungWei Chiu <[email protected]>
alexellis pushed a commit to alexellis/arkade that referenced this issue Apr 4, 2022
- No official arm64 build for now but the current amd64 binary works for the M1 env, should change to the arm64 binary once jq has official arm64 version in its Github release.
- Reference Github issue: jqlang/jq#2386

Signed-off-by: HungWei Chiu <[email protected]>
@balupton
Copy link

balupton commented May 3, 2022

On my mac mini m1, which is running macos in a parallels virtual machine, the https://github.com/stedolan/jq/releases/download/jq-1.6/jq-osx-amd64 binary is failing with:

balupton@Benjamins-Virtual-Machine ~ % ./.local/bin/jq --help            
zsh: bad CPU type in executable: ./.local/bin/jq

I tried using arch to trick it, but the same deal:

balupton@Benjamins-Virtual-Machine ~ % arch -x86_64 ./.local/bin/jq --help
arch: posix_spawnp: ./.local/bin/jq: Bad CPU type in executable

Here's the arch details if that helps:

balupton@Benjamins-Virtual-Machine ~ % uname -m
arm64
balupton@Benjamins-Virtual-Machine ~ % uname -a
Darwin Benjamins-Virtual-Machine.local 21.4.0 Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:19 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_VMAPPLE arm64

@balupton
Copy link

balupton commented May 3, 2022

Okay installing rosetta via the CLI solved it:

/usr/sbin/softwareupdate --install-rosetta --agree-to-license

alexellis pushed a commit to alexellis/arkade that referenced this issue Jun 29, 2022
- No official arm64 build for now but the current amd64 binary works for the M1 env, should change to the arm64 binary once jq has official arm64 version in its Github release.
- Reference Github issue: jqlang/jq#2386

Signed-off-by: HungWei Chiu <[email protected]>
@magnetikonline
Copy link

you can use arm 1.6 binary from https://github.com/slamdev/rules_jq/releases/download/0.0.1/jq-osx-arm64 until the official build is released.

Thanks @slamdev - sadly it's not statically compiled.

$ jq --version
dyld[2211]: Library not loaded: /opt/homebrew/opt/oniguruma/lib/libonig.5.dylib
  Referenced from: /usr/local/bin/jq
  Reason: tried: '/opt/homebrew/opt/oniguruma/lib/libonig.5.dylib' (no such file), '/usr/local/lib/libonig.5.dylib' (no such file), '/usr/lib/libonig.5.dylib' (no such file)
Abort trap: 6

I don't use homebrew.

@magnetikonline
Copy link

Okay - got inspired to cobble up a solution to build jq for M1/arm without the need for Homebrew.

https://gist.github.com/magnetikonline/58eb344e724d878345adc8622f72be13

Installs autoconf/automake/libtool - downloads jq 1.6 tarball, patches as needed and compiles/installs.

Result:

$ file /usr/local/bin/jq
/usr/local/bin/jq: Mach-O 64-bit executable arm64

Tested on MacBook Apple M1 Pro under Monterey 12.4.

DanielLiebesk added a commit to DanielLiebesk/arkade that referenced this issue Aug 24, 2022
- No official arm64 build for now but the current amd64 binary works for the M1 env, should change to the arm64 binary once jq has official arm64 version in its Github release.
- Reference Github issue: jqlang/jq#2386

Signed-off-by: HungWei Chiu <[email protected]>
@n8felton n8felton moved this to 6️⃣4️⃣ Apple Silicon (arm64) support in @n8felton's backlog Nov 4, 2022
@HWiese1980
Copy link

What's the status here? Are we going to see an official M1 release? Is one in the making? I've tried installing it via asdf-vm, to no avail. It states that, Sadly, there are no official releases for your architecture

@jonathan-klein-da
Copy link

Still no movement here?

@uogbuji
Copy link

uogbuji commented Jun 7, 2023

Just a +1 here. I'm another one who never plans to use homebrew, and I can find M1-compatible installers for most software I use, except for jq.

@reneleonhardt
Copy link

Is this not doable? I can't debug the GitHub workflow with act, but building on M1 shows 7 tests passed.

$ ARCH=x86_64
$ ./configure --with-oniguruma=builtin --host=$ARCH-apple-darwin$(uname -r) CC="gcc -arch $ARCH"
$ make clean && make -j 8 && (make check; echo "$(file ./jq) built against host $ARCH-apple-darwin$(uname -r)")
./jq: Mach-O 64-bit executable x86_64 built against host x86_64-apple-darwin22.5.0

$ ARCH=arm64
$ ./configure --with-oniguruma=builtin --host=$ARCH-apple-darwin$(uname -r) CC="gcc -arch $ARCH"
$ make clean && make -j 8 && (make check; echo "$(file ./jq) built against host $ARCH-apple-darwin$(uname -r)")
./jq: Mach-O 64-bit executable arm64 built against host arm64-apple-darwin22.5.0

$ gcc --version
Apple clang version 14.0.3 (clang-1403.0.22.14.1)
Target: arm64-apple-darwin22.5.0

reneleonhardt added a commit to reneleonhardt/jq that referenced this issue Jun 17, 2023
@itchyny itchyny added this to the 1.7 release milestone Jun 25, 2023
@uogbuji
Copy link

uogbuji commented Jul 5, 2023

@reneleonhardt sorry, I missed your response, and thanks. The problem is that jq comes with configure.ac, not configure, so your recipe doesn't work standalone. I was hoping not to have to wrestle with building autotools & autoconf from source. I think that's a vain hope however, and I I'm going to just write up a gist on installing those for people like me who are allergic to homebrew & macports. Once I do that, I suspect your recipe will work just fine.

@wader
Copy link
Member

wader commented Jul 5, 2023

@uogbuji Hi, versioned release like jq-1.6.tar.gz should include configure but confusingly "Source code (tar.gz)" on the release page does not, that is a packaging of the git repo.

$ curl -sL https://github.com/jqlang/jq/archive/refs/tags/jq-1.6.tar.gz | tar tz | grep configure
jq-jq-1.6/configure.ac
$ curl -sL https://github.com/jqlang/jq/releases/download/jq-1.6/jq-1.6.tar.gz | tar tz | grep configure
jq-1.6/configure
jq-1.6/configure.ac
jq-1.6/modules/oniguruma/configure.ac

@uogbuji
Copy link

uogbuji commented Jul 5, 2023

Hi, versioned release like jq-1.6.tar.gz should include configure but confusingly "Source code (tar.gz)" on the release page does not, that is a packaging of the git repo.

Ah. Helpful; thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet