Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ruby 3.2.2 + Multi-Arch CI Builds #133

Closed
wants to merge 31 commits into from
Closed

Ruby 3.2.2 + Multi-Arch CI Builds #133

wants to merge 31 commits into from

Conversation

YOU54F
Copy link

@YOU54F YOU54F commented Apr 19, 2023

Continuation from #132

This PR

  • Upgrades to 3.2.2
  • Provides various CI platforms (GH Actions / Cirrus CI / CircleCI)

It is a bit a WIP and needs cleaning up, plus I have a couple of questions / outstanding hacks to resolve, but at least gives a place for others to review :)

Upgrade to Ruby 3.1.2 / Support ARM64 OSX/Linux

Hey Hey @FooBarWidget 👋🏾

We are the maintainers of Pact! You might remember us from https://www.joyfulbikeshedding.com/blog/2021-01-06-the-future-of-traveling-ruby.html

Traveling Ruby hasn't seen updates for quite a while now, but it's still being used. Yesterday, the author of the Pact contract testing framework asked me to do a new release, because he needed a bug fix for supporting paths that contain spaces. So I gave it a try. Predictably, the build system is broken due to all the changes in Linux and macOS the past few years, and fixing it all takes a lot of effort.

We have since moved onto using Rust to build and package most of core functionality, but still get a-lot of asks to update the build system we use for packaging our Pact Ruby implementation, and its various gems into a standalone package, with travelling ruby.

I took a look a few times, but felt out of my depth, but have been watching a few forks closely on an off over the years, and finally managed to pul some work together from others, and added some work myself.

Issues affecting our community

Write a description of changes made

  • Update to Ruby 3.1.2 75afdff
  • Support for Arm64 Darwin b5fb0ad
  • Support for both Arm64 & x86_64 builds 4fd8c3a
  • Support for aarch64 on linux ff9c059
  • Upgrade to Ruby 3.2.2
  • Add CI Support for
    • GitHub for all supported hosted operating systems/arches
    • CircleCI for ARM64 Linux
    • CirrusCI for ARM64 OSX
  • Windows
    • Provide builds for both x86 and x86_64 as both available, and some users have issues with ucrt and need msvs version of Ruby (Windows on ARM users)

BONUS POINTS

  • Check if the commit's or even all commits' message styles matches ours.
  • Extend the repo's README / documentation, if applicable.

Avoid breakage

  • Pull requests should normally be submitted against the latest stable branch (e.g. main).

👍🏾

  • Elaborate how you've tested your code.

Release for 3.2.2

https://github.com/YOU54F/traveling-ruby/releases/tag/rel-20230508

Travelling ruby 3.2.2

OS Ruby Architecture Supported
OSX 3.2.2 x86_64
OSX 3.2.2 aarch64 (arm)
Linux 3.2.2 x86_64
Linux 3.2.2 aarch64 (arm)
Windows 3.2.2 x86_64
Windows 3.2.2 x86
Windows 3.2.2 aarch64 (via x86 emulation)

Changes tested and released against a fork of pact-ruby-standalone project

https://github.com/YOU54F/pact-ruby-standalone/releases

You can see the traveling ruby binaries being pulled in here

https://github.com/YOU54F/pact-ruby-standalone/blob/master/tasks/package.rake#L218

Packages tested cross platform in CI

https://github.com/YOU54F/pact-ruby-standalone/actions/runs/4671665215

and consumed in pact-js (a heavily used project which consumed the ruby standalone project)

https://github.com/YOU54F/pact-js-core/blob/pact-node-arm64/standalone/install.ts#L28-L29

It previously wasn't able to run on arm64 linux machines, and on darwin had to use rosetta, I've now tested that without Rosetta installed, our ruby executables are looking good so far.

Consumed here for testing against pact-python pact-foundation/pact-python#342

If this is a relatively large or complex change, kick off the discussion by explaining why you chose the solution you did and what alternatives you considered, etc...

I think I would still like to use the squashfs approach used in ruby-packer, I did tickle both their source codes a little bit, but the forks were more progressed against yours, and it has been tried and tested with our applications for years.

@YOU54F
Copy link
Author

YOU54F commented Apr 19, 2023

You can run the .cirrus.yml locally on a mac arm based machine with

https://tart.run/ and https://github.com/cirruslabs/cirrus-cli

brew install cirruslabs/cli/tart
tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
brew install cirruslabs/cli/cirrus

Then type cirrus run

You can then test with rosetta installed, and without

by using softwareupdate --install-rosetta --agree-to-license within the virtual machine, so nice way to test that a system can run with only arm64 binaries (most of us will have rosetta on our dev machines) - tart gives you a nice ephemeral clean slate

@utoppo
Copy link

utoppo commented May 3, 2023

You can run the .cirrus.yml locally on a mac arm based machine with

https://tart.run/ and https://github.com/cirruslabs/cirrus-cli

brew install cirruslabs/cli/tart
tart clone ghcr.io/cirruslabs/macos-ventura-base:latest ventura-base
brew install cirruslabs/cli/cirrus

Then type cirrus run

You can then test with rosetta installed, and without

by using softwareupdate --install-rosetta --agree-to-license within the virtual machine, so nice way to test that a system can run with only arm64 binaries (most of us will have rosetta on our dev machines) - tart gives you a nice ephemeral clean slate

Unfortunately this does not work for me. (tested on Mac mini M1, 2020, macOS Ventura 13.3.1 and MBA 2020, also macOS Ventura 13.3.1)
Output of macosx-arm64-' task:

❌ 'macosx-arm64-' task 59s
   ✅ pull virtual machine 0.8s
   ✅ clone virtual machine 0.1s
   ✅ boot virtual machine 10s
   ✅ syncing working directory 2.6s
   ❌ 'main' script 33s
      rbenv local system
      sudo gem install bundler:2.3.17
      Successfully installed bundler-2.3.17
      Parsing documentation for bundler-2.3.17
      Installing ri documentation for bundler-2.3.17
      Done installing documentation for bundler after 0 seconds
      1 gem installed
      rbenv global 3.2.2
      cd osx
      rake stash_conflicting_paths
      rake --trace
      ** Invoke default (first_time)
      ** Invoke package (first_time)
      ** Invoke package:3.2.2 (first_time)
      ** Invoke traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz (first_time)
      ** Invoke output/3.2.2/bin (first_time)
      ** Invoke runtime/ok (first_time)
      ** Invoke setup-runtime (first_time, not_needed)
      ** Execute runtime/ok
      ./setup-runtime runtime
      rake aborted!
      Command failed with status (127): [./setup-runtime runtime...]
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/file_utils.rb:67:in `block in create_shell_runner'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/file_utils.rb:57:in `sh'
      /private/tmp/cirrus-ci/working-dir/osx/Rakefile:121:in `block in <top (required)>'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
      /usr/bin/rake:23:in `load'
      /usr/bin/rake:23:in `<main>'
      Tasks: TOP => default => package => package:3.2.2 => traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz => output/3.2.2/bin => runtime/ok

@utoppo
Copy link

utoppo commented May 3, 2023

Some files seem not to be executable from within the virtual machine (at least for me):

local:

utoppo@m1mini osx % ls -la
total 104
drwxr-xr-x   9 utoppo  staff    288 May  3 10:05 .
drwxr-xr-x@ 31 utoppo  staff    992 May  3 14:20 ..
-rw-r--r--   1 utoppo  staff   1783 May  3 10:05 README.md
-rw-r--r--   1 utoppo  staff   3760 May  3 10:05 Rakefile
-rwxr-xr-x   1 utoppo  staff  11921 May  3 10:05 build-ruby
drwxr-xr-x  10 utoppo  staff    320 May  3 10:05 internal
-rwxr-xr-x   1 utoppo  staff    113 May  3 10:05 package
-rwxr-xr-x   1 utoppo  staff  24064 May  3 10:05 setup-runtime
-rwxr-xr-x   1 utoppo  staff    115 May  3 10:05 test-gems

virtual machine:

total 104
drwxr-xr-x   9 admin  wheel    288 May  3 12:33 .
drwxr-xr-x  32 admin  wheel   1024 May  3 12:34 ..
-rw-r--r--   1 admin  wheel   1783 May  3 12:33 README.md
-rw-r--r--   1 admin  wheel   3760 May  3 12:33 Rakefile
-rw-r--r--   1 admin  wheel  11921 May  3 12:33 build-ruby
drwxr-xr-x  10 admin  wheel    320 May  3 12:33 internal
-rw-r--r--   1 admin  wheel    113 May  3 12:33 package
-rw-r--r--   1 admin  wheel  24064 May  3 12:33 setup-runtime
-rw-r--r--   1 admin  wheel    115 May  3 12:33 test-gems

Adding the following at the beginning of themacosx-arm64-task inside cirrus.yml seems to help:

    - find osx/ -type f -exec chmod +x {} \;
    - chmod +x shared/package

@YOU54F
Copy link
Author

YOU54F commented May 3, 2023

ahh sorry yeah that is a known issue and their is a workaround which is to run with --dirty flag or chmod the files as file perms aren't copied

https://github.com/cirruslabs/cirrus-cli#running-cirrus-tasks

Related issues

cirruslabs/cirrus-cli#528

It allows to relation to this PR

cirruslabs/cirrus-cli#526

where one can retrieve the artefacts generated from the VM

@YOU54F
Copy link
Author

YOU54F commented May 3, 2023

Thanks for taking a look!

Have you had a chance to try it on your m1 machine without cirrus?

It runs pretty quickly, think it was about 10 minutes for me

@FooBarWidget
Copy link
Member

I'll also take a look as soon as I can.

@YOU54F
Copy link
Author

YOU54F commented May 3, 2023

No rush! I think I may have an issue with libyaml and psych.so which affects linux users that don't have libyaml installed

See this PR against our consuming project, pact-ruby-standalone

pact-foundation/pact-ruby-standalone#103

@utoppo
Copy link

utoppo commented May 4, 2023

ahh sorry yeah that is a known issue and their is a workaround which is to run with --dirty flag or chmod the files as file perms aren't copied

https://github.com/cirruslabs/cirrus-cli#running-cirrus-tasks

Related issues

cirruslabs/cirrus-cli#528

It allows to relation to this PR

cirruslabs/cirrus-cli#526

where one can retrieve the artefacts generated from the VM

Thanks for your reply! Using the --dirty flag did not work for me. See output below.

Below is the output when running cirrus run --dirty locally:

utoppo@m1mini traveling-ruby % cirrus run --dirty
🕔 'macosx-arm64-' task 43s
   ✅ pull virtual machine 0.6s
   ✅ clone virtual machine 0.1s
   ✅ boot virtual machine 6.2s
   🕔 'main' script 31s
      ...
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
      /usr/bin/rake:23:in `load'
      /usr/bin/rake:23:in `<main>'
      Tasks: TOP => default => package => package:3.2.2 => traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz => output/3.2.2/bin => runtime/ok
      
      ** Invoke package:3.2.2 (first_time)
      ** Invoke traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz (first_time)
      ** Invoke output/3.2.2/bin (first_time)
      ** Invoke runtime/ok (first_time)
      ** Invoke setup-runtime (first_time, not_needed)
      ** Execute runtime/ok
      ./setup-runtime runtime
      Initializing...
      ------------------------------------------
      + mkdir -p /Volumes/My Shared Files/working-dir/osx/runtime
      + mkdir -p /Volumes/My Shared Files/working-dir/osx/runtime/ccache
      Entering /Volumes/My Shared Files/working-dir/osx/runtime
      
      Installing tool 1/6: CMake...
      ------------------------------------------
      + rm -f cmake-3.23.2.tar.gz
      + curl --fail -L -o cmake-3.23.2.tar.gz https://github.com/Kitware/CMake/releases/download/v3.23.2/cmake-3.23.2.tar.gz
        %!T(MISSING)otal    %!R(MISSING)eceived %!X(MISSING)ferd  Average Speed   Time    Time     Time  Current
      
                                       Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0
       --:--:-- --:--:
      -- --:--:--     0
  0     0    0     0    0     0      0      0 --:-
      -:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  4 9753k    4  463k    0     0   396k      0  0:00:24  0:00:01  0:00:23  396k
 66 9753k   66 6514k    0     0  2996k      0  0:00:03  0:00:02  0:00:01 6032k
100 9753k  100 9753k    0     0  4010k      0  0:00:02  0:00:02 --:--:-- 7366k
      + tar xzf cmake-3.23.2.tar.gz
      + rm cmake-3.23.2.tar.gz
      Entering /Volumes/My Shared Files/working-dir/osx/runtime/cmake-3.23.2
      Removing some false positive checks in CMakeLists
      patch: **** too many file arguments
      rake aborted!
      Command failed with status (2): [./setup-runtime runtime...]
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/file_utils.rb:67:in `block in create_shell_runner'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/file_utils.rb:57:in `sh'
      /Volumes/My Shared Files/working-dir/osx/rakefile:121:in `block in <top (required)>'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `block in execute'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:273:in `execute'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:214:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:238:in `block in invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:236:in `invoke_prerequisites'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
      /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib/ruby/2.6.0/monitor.rb:235:in `mon_synchronize'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:194:in `invoke_with_call_chain'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/task.rb:183:in `invoke'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:160:in `invoke_task'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block (2 levels) in top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `each'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:116:in `block in top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:125:in `run_with_threads'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:110:in `top_level'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:83:in `block in run'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:186:in `standard_exception_handling'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/lib/rake/application.rb:80:in `run'
      /Library/Ruby/Gems/2.6.0/gems/rake-12.3.3/exe/rake:27:in `<top (required)>'
      /usr/bin/rake:23:in `load'
      /usr/bin/rake:23:in `<main>'
      Tasks: TOP => default => package => package:3.2.2 => traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz => output/3.2.2/bin => runtime/ok
      
Error: build failed: task macosx-arm64- (0) failed
2023/05/04 12:25:58 build failed: task macosx-arm64- (0) failed

@utoppo
Copy link

utoppo commented May 4, 2023

Thanks for taking a look!

Have you had a chance to try it on your m1 machine without cirrus?

It runs pretty quickly, think it was about 10 minutes for me

The arm-build-task takes ~15-20 minutes on my M1-machines (Mac Mini 2020 16GB, MBA 2020 8GB).

utoppo@m1mini traveling-ruby % cirrus run                            
✅ 'macosx-arm64-' task 15:15

Here's a screenshot from cirrus-ci.com. (I had to merge the ci-branch into main for anything to happen over there)
image

This may sound like a stupid question: Where can i find the binaries after the build has finished? Both locally and on cirrus-ci.com? Using the following flag locally didn't seem to have any effect:

cirrus run --artifacts-dir artifacts

@YOU54F
Copy link
Author

YOU54F commented May 4, 2023

I've been running the arm64 macos builds on an m1 pro / 16gb, so might have a bit more legs than the mini and the air

I haven't trialled that out yet but its a combination of

https://cirrus-ci.org/guide/writing-tasks/#artifacts-instruction

and the --artifacts-dir option

--artifacts-dir string directory in which to save the artifacts

from cirrus run --help

Execute Cirrus CI tasks locally

Usage:
  cirrus run [flags] [task]

Flags:
      --affected-files strings             comma-separated list of files to add to the list of affected files (used in changesInclude and changesIncludeOnly functions)
      --affected-files-git string          Git revision (e.g. HEAD, v0.1.0 or commit SHA) to compare unstaged changes against and add changed files to the list of affected files (similarly to git diff)
      --affected-files-git-cached string   Git revision (e.g. HEAD, v0.1.0 or commit SHA) to compare staged changes against and add changed files to the list of affected files (similarly to git diff --cached)
      --artifacts-dir string               directory in which to save the artifacts
      --container-backend string           container engine backend to use, either "docker", "podman" or "auto" (default "auto")
      --container-lazy-pull                attempt to pull images only if they are missing locally (helpful in case of registry rate limits)
      --dirty                              if set the project directory will be mountedin read-write mode, otherwise the project directory files are copied, taking .gitignore into account
      --dockerfile-image-push              whether to push whe image produced by the Dockerfile as CI environment feature
      --dockerfile-image-template string   image that Dockerfile as CI environment feature should produce (default "gcr.io/cirrus-ci-community/%s:latest")
  -e, --environment stringArray            set (-e A=B) or pass-through (-e A) an environment variable
  -h, --help                               help for run
      --lazy-pull                          attempt to pull container and VM images only if they are missing locally (helpful in case of registry rate limits; enables --container-lazy-pull and --tart-lazy-pull)
  -o, --output string                      output format of logs, supported values: auto, interactive, noemoji, simple, travis, github-actions, teamcity (default "auto")
      --tart-lazy-pull                     attempt to pull Tart VM images only if they are missing locally (helpful in case of registry rate limits)
  -v, --verbose

@YOU54F
Copy link
Author

YOU54F commented May 4, 2023

I stand corrected - 14mins 28 on my machine

touch traveling-ruby-gems-20230428-3.2.2-osx-arm64/ok
rake  1417.79s user 488.49s system 219% cpu 14:27.98 total

traveling-ruby/osx on  
ci [$] on ☁️  (eu-west-2) took 14m28s
🕙13:26:48 ❯

Screenshot 2023-05-04 at 13 28 06

@utoppo
Copy link

utoppo commented May 5, 2023

The following did the trick for me, to get the built files, locally:

.cirrus.yml:

...
macosx-arm64-task:
  timeout_in: 90m
  macos_instance:
    image: ghcr.io/cirruslabs/macos-ventura-base:latest
  script:
    - find osx/ -type f -exec chmod +x {} \;
    - chmod +x shared/package
    - mkdir -p "build"
    - rbenv local system
    - sudo gem install bundler:2.3.17
    - rbenv global 3.2.2
    - cd osx
    - rake stash_conflicting_paths
    - rake --trace
    - cp -R traveling-ruby* ../build
  binary_artifacts:
    path: "build/*"
...

then run:
cirrus run --artifacts-dir build

@utoppo
Copy link

utoppo commented May 8, 2023

The change in .cirrus.yml also leads to changes on cirrus-ci.com, when pushed to the repository. After a successful build, a new section "artifacts" appears inside the task, where you can download traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz.

How could I resolve the "skipping uploading of folder"-message? Is the gems-folder traveling-ruby-gems-20230428-3.2.2-osx-arm64 somehow needed for using traveling ruby? Or does traveling-ruby-20230428-3.2.2-osx-arm64.tar.gz contain everything one needs, including the gems?

Skipping uploading of '/var/folders/dc/nt69f7cn7r5bgp90zkhs00000000gn/T/cirrus-ci-build/build/traveling-ruby-gems-20230428-3.2.2-osx-arm64' because it's a folder

image

@YOU54F
Copy link
Author

YOU54F commented May 30, 2023

Hey @utoppo

You need to upload just archived files, so *.tar.gz

  binary_artifacts:
    path: "linux/*.tar.gz"

@YOU54F
Copy link
Author

YOU54F commented May 30, 2023

Hey

Just as a heads-up, I've been working on another branch next

https://github.com/YOU54F/traveling-ruby/releases/tag/rel-20230529-next

I found issues with libyaml and libffi, so building from source in the Docker image in linux.

Also making some improvements to allow for building and testing multiple ruby versions, the ruby gems, across multiple platforms, archs and docker image variants

Docker emulation means you can build/test cross arch for the linux binaries, and cross plat on a mac.

mac m1 runners mean we can test with with rosetta and without, and some changes to the macos build suite will allow for arm64 builds on x86_64 runners

looking to build a custom image with the runtime prebuilt for macos, to speed up execution :)

it's like pokemon, gotta build em all

Screenshot 2023-05-30 at 18 02 30

@dbackeus
Copy link

@YOU54F loving to follow your work on this ❤️

Once complete it will be a huge step forward for ruby executable portability. Looking forward to making use of it!

@YOU54F
Copy link
Author

YOU54F commented Jun 7, 2023

ARM64 + x86_64 builds for 2.6 / 3.0.x / 3.1.x / 3.2.x & 3.3.x-preview

https://github.com/YOU54F/traveling-ruby/releases/tag/rel-20230605

Wrapper scripts are now sh compliant.

osx builds, now half-sized, (they had an extra .a binary)

all of the existing gems built, bar mysql2.

https://github.com/YOU54F/traveling-ruby

Traveling-Ruby. Done 🚀

@YOU54F
Copy link
Author

YOU54F commented Aug 3, 2023

Closing in favour of #134

Latest release has just gone out

https://github.com/YOU54F/traveling-ruby/releases/tag/rel-20230803

@YOU54F YOU54F closed this Aug 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants