Fails to load with Rubygems 3.4 #56

davekaro · 2022-12-25T16:45:48Z

I've installed ruby 3.2 on macOS Ventura (13.0.1) with M1 (arm64) using ruby-build. Launching irb and typing require "argon2" I see the following:

/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:145:in `block in ffi_lib': Could not open library '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle': dlopen(/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle, 0x0005): tried: '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (unloadable mach-o file type 10 '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle'), '/System/Volumes/Preboot/Cryptexes/OS/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (no such file), '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle.dSYM/Contents/Resources/DWARF/libargon2_wrap.bundle' (unloadable mach-o file type 10 '/Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/arg (LoadError)
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `map'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-1.15.5/lib/ffi/library.rb:99:in `ffi_lib'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:10:in `<module:Ext>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:8:in `<module:Argon2>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:6:in `<top (required)>'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/lib/argon2.rb:4:in `<top (required)>'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:160:in `rescue in require'
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:149:in `require'
	from (irb):2:in `<main>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `load'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `<main>'
<internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require': cannot load such file -- argon2 (LoadError)
	from <internal:/Users/dave/.rbenv/versions/3.2.0/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
	from (irb):2:in `<main>'
	from /Users/dave/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/irb-1.6.2/exe/irb:11:in `<top (required)>'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `load'
	from /Users/dave/.rbenv/versions/3.2.0/bin/irb:25:in `<main>'

Running on CI with Github Actions and ruby 3.2 on Ubuntu 22.04.1 (x86_64 arch) I see a similar but different error:

LoadError: cannot find 'argon2_wrap' library
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:10:in `<module:Ext>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:8:in `<module:Argon2>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2/ffi_engine.rb:6:in `<main>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/argon2-2.1.2/lib/argon2.rb:4:in `<main>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
/home/runner/work/my-project-name/my-project-name/config/application.rb:24:in `<top (required)>'
/home/runner/work/my-project-name/my-project-name/config/environment.rb:2:in `require_relative'
/home/runner/work/my-project-name/my-project-name/config/environment.rb:2:in `<top (required)>'
/home/runner/work/my-project-name/my-project-name/Rakefile:3:in `require_relative'
/home/runner/work/my-project-name/my-project-name/Rakefile:3:in `block in <top (required)>'
/home/runner/work/my-project-name/my-project-name/vendor/bundle/ruby/3.2.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/opt/hostedtoolcache/Ruby/3.2.0/x64/bin/bundle:25:in `load'
/opt/hostedtoolcache/Ruby/3.2.0/x64/bin/bundle:25:in `<main>'
Tasks: TOP => assets:precompile => environment
(See full trace by running task with --trace)

I tried cloning the argon2 repo, and was able to run bin/setup, bin/test, and rake test all without any errors. I looked at https://github.com/technion/ruby-argon2/blob/master/test/test_helper.rb#L3 and found this helped. Steps to reproduce:

Open irb session in the ruby-argon2 directory
Type require "argon2" see failures (same as above for macOS)
Type $LOAD_PATH.unshift File.expand_path("lib")
Type require "argon2" and now it works.

Note the above only works after running bin/setup, even though I've installed argon2 already. This seems to be because when running bin/setup, the correct file in ext/argon2_wrap is created, specifically the libargon_wrap2 file:

On macOS, Ruby 3.1.3 (argon2 loads no issue)

~ ❯❯❯ ls -la .rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 496
drwxr-xr-x  9 dave  staff     288 Dec  5 16:58 .
drwxr-xr-x  4 dave  staff     128 Dec  5 16:58 ..
-rw-r--r--  1 dave  staff    1958 Dec  5 16:58 Makefile
-rw-r--r--  1 dave  staff    4504 Dec  5 16:58 argon_wrap.c
-rw-r--r--  1 dave  staff      98 Dec  5 16:58 extconf.rb
-rwxr-xr-x  1 dave  staff   76450 Dec  5 16:58 libargon2_wrap.bundle
drwxr-xr-x  3 dave  staff      96 Dec  5 16:58 libargon2_wrap.bundle.dSYM
-rwxr-xr-x  1 dave  staff  151552 Dec  5 16:58 libargon2_wrap.so
-rw-r--r--  1 dave  staff    4500 Dec  5 16:58 test.c

On macOS, Ruby 3.2.0 (missing the libargon2_wrap.bundle file, argon2 errors on require)

~ ❯❯❯ ls -la .rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 344
drwxr-xr-x  8 dave  staff     256 Dec 25 12:14 .
drwxr-xr-x  4 dave  staff     128 Dec 25 12:14 ..
-rw-r--r--  1 dave  staff    1958 Dec 25 12:14 Makefile
-rw-r--r--  1 dave  staff    4504 Dec 25 12:14 argon_wrap.c
-rw-r--r--  1 dave  staff      98 Dec 25 12:14 extconf.rb
drwxr-xr-x  3 dave  staff      96 Dec 25 12:14 libargon2_wrap.bundle.dSYM
-rwxr-xr-x  1 dave  staff  151552 Dec 25 12:14 libargon2_wrap.so
-rw-r--r--  1 dave  staff    4500 Dec 25 12:14 test.c

On Ubuntu, Ruby 3.1.3 (argon2 loads no issue)

$ ls -la .rbenv/versions/3.1.3/lib/ruby/gems/3.1.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 256
drwxrwxr-x 2 ubuntu ubuntu   4096 Dec 25 18:10 .
drwxrwxr-x 4 ubuntu ubuntu   4096 Dec 25 18:10 ..
-rw-r--r-- 1 ubuntu ubuntu   1958 Dec 25 18:10 Makefile
-rw-r--r-- 1 ubuntu ubuntu   4504 Dec 25 18:10 argon_wrap.c
-rw-r--r-- 1 ubuntu ubuntu     98 Dec 25 18:10 extconf.rb
-rwxrwxr-x 1 ubuntu ubuntu 227056 Dec 25 18:10 libargon2_wrap.so
-rw-r--r-- 1 ubuntu ubuntu   4500 Dec 25 18:10 test.c

On Ubuntu, Ruby 3.2.0 (missing the libargon2_wrap.so file, argon2 errors on require)

$ ls -la .rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.2/ext/argon2_wrap/
total 32
drwxrwxr-x 2 ubuntu ubuntu 4096 Dec 25 17:49 .
drwxrwxr-x 4 ubuntu ubuntu 4096 Dec 25 17:49 ..
-rw-r--r-- 1 ubuntu ubuntu 1958 Dec 25 17:49 Makefile
-rw-r--r-- 1 ubuntu ubuntu 4504 Dec 25 17:49 argon_wrap.c
-rw-r--r-- 1 ubuntu ubuntu   98 Dec 25 17:49 extconf.rb
-rw-r--r-- 1 ubuntu ubuntu 4500 Dec 25 17:49 test.c

I'm not sure if this is an ffi issue, or the way argon2 is using ffi.

The text was updated successfully, but these errors were encountered:

scott-knight · 2022-12-25T19:22:21Z

Found the same issue this morning when updating my app during deploy test to github. Ruby 3.2.0, Rails 7.0.4, Ubuntu 22.04 instance

communiteq · 2022-12-25T19:35:01Z

Same issue here. Ruby 2.7.6p219. Broke since this morning (5 AM UTC).

technion · 2022-12-26T08:52:38Z

Thanks for reporting this and the detail. Much like the work in this thread, I'm struggling to work out whether this is an FFI issue or just something involving the build scripts. I'm working with the Ruby 3.2-rc1 Docker image (which currently seems to be the latest) and I can replicate it there.

I'm going through open issues on FFI and nothing seems relevant.

I've found if you make sure the library is built and do an strace, it does load it even before complaining it's not built:

openat(AT_FDCWD, "/ruby-argon2/ext/argon2_wrap/libargon2_wrap.so", O_RDONLY|O_CLOEXEC) = 5

Whatever's going on I'm sure someone else's gem will have the same issue, which I'm hoping leads to a fix.

@communiteq Just to confirm are you saying you had this same break without upgrading to Ruby 3.2.0? I can't replicate that.

technion · 2022-12-26T09:01:31Z

@davekaro Regarding this step you supplied:

Type $LOAD_PATH.unshift File.expand_path("lib")

You may notice this is actually called in the Gemfile already:

https://github.com/technion/ruby-argon2/blob/master/argon2.gemspec#L5

I wonder if Ruby 3.2 changed the behaviour around recognising this?

technion · 2022-12-26T09:25:40Z

You know I went through the FFI changes and couldn't find anything that looked relevant, but I bumped from 1.14 to 1.15 and CI just gave an all pass with 3.2 now tested.

I will cut a new version as soon as I feed my cats.

technion · 2022-12-26T09:36:00Z

Done. Can anyone else check with version 2.1.3 and confirm?

communiteq · 2022-12-26T10:29:57Z

Yes, confirmed, tests suddenly started breaking yesterday. And worse, updating to 2.1.3 does not resolve the issue.
Let me know if there is anything you need to know.

ruby 2.7.6p219 (2022-04-12 revision c9c2245c0a) [x86_64-linux]
Ubuntu 22.04.1 LTS

Not happening on Ubuntu 18.04.6 LTS

The gem is loaded as part of a Discourse plugin. Something must have changed somewhere... (and it was not the plugin)

gem install ffi -v 1.15.5 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Building native extensions. This could take a while...
Successfully installed ffi-1.15.5
1 gem installed
gem install ffi-compiler -v 1.0.1 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Successfully installed ffi-compiler-1.0.1
1 gem installed
gem install argon2 -v 2.1.3 -i /var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6 --no-document --ignore-dependencies --no-user-install
Building native extensions. This could take a while...
Successfully installed argon2-2.1.3
1 gem installed
rake aborted!
LoadError: cannot find 'argon2_wrap' library
/var/www/discourse/plugins/discourse-migratepassword/gems/2.7.6/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find'

technion · 2022-12-26T10:36:43Z

@communiteq If you're running 2.7 I'm sure you have a different issue. I do know that I've walked over to multiple platforms, installed the 2.1.3 gem and then used it from irb without issue. I'm happy to see if there's anything I can do there but in the meantime, can either of the people who broke on 3.2 advise?

communiteq · 2022-12-26T10:45:52Z

Given the timing of "both" issues I do suspect that they have the same root cause. So yes, I absolutely understand that the issue is now resolved from your side. I am just hoping that someone ever finds out what has caused this in the first place, since that might give me a clue to working towards a resolution. It's surprising that bumping the requirements to higher versions worked for you since I was already on ffi-1.15.5 and ffi-compiler-1.0.1. Weird huh?

That said, the Argon2 gem is (somehow!) the only gem that has broken in this pretty complex install.

technion · 2022-12-26T10:50:39Z

I do see your point, I'm just stuck on the fact there haven't been any pushes to this repo for some time, so outside of Ruby 3.2 I can't see what would break your build. I do know that looking at the module you refer to:

https://github.com/communiteq/discourse-migratepassword/blob/master/plugin.rb

Use of 2.1.1 seems hardcoded there, so I don't believe it's going to matter what you happen to have installed.

technion · 2022-12-26T11:39:34Z

Just to confirm, it appears that although the CI works fine, there's something still broken here. I had a fair bit of trouble due to the lack of a current Ruby 3.2 docker container but I've built a more complete lab of this.

root@7e9fcf33fcaf:/# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

# Install prereqs..

eval "$(/root/.rbenv/bin/rbenv init - bash)"

rbenv install 3.2.0
root@7e9fcf33fcaf:/# mkdir myapp
root@7e9fcf33fcaf:/# cd myapp
root@7e9fcf33fcaf:/myapp# bundle init
Writing new Gemfile to /myapp/Gemfile
root@7e9fcf33fcaf:/myapp# nano Gemfile
root@7e9fcf33fcaf:/myapp# cat Gemfile
# frozen_string_literal: true

source "https://rubygems.org"
gem "argon2", "2.1.3"
root@7e9fcf33fcaf:/myapp# bundle install
Don't run Bundler as root. Installing your bundle as root will break this application for all non-root users on this
machine.
Fetching gem metadata from https://rubygems.org/.
Using rake 13.0.6
Fetching ffi 1.15.5
Using bundler 2.4.1
Installing ffi 1.15.5 with native extensions
Fetching ffi-compiler 1.0.1
Installing ffi-compiler 1.0.1
Fetching argon2 2.1.3
Installing argon2 2.1.3 with native extensions
Bundle complete! 1 Gemfile dependency, 5 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
root@7e9fcf33fcaf:/myapp# bundle exec myapp.rb
bundler: failed to load command: myapp.rb (myapp.rb)
/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'argon2_wrap' library (LoadError)

I'll continue investigating but I'm sure appreciative of any other suggestions. I do note that in this environment, it simply seems to never be running the actual build script.

openat(AT_FDCWD, "/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 5
openat(AT_FDCWD, "/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap", O_RDONLY|O_CLOEXEC|O_DIRECTORY) = 5
write(2, "/root/.rbenv/versions/3.2.0/lib/"..., 2163/root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'argon2_wrap' library (LoadError)
root@7e9fcf33fcaf:/myapp# ls /root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap
Makefile  argon_wrap.c  extconf.rb  test.c

This workaround takes care of it:

root@7e9fcf33fcaf:/myapp# cd /root/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap
root@7e9fcf33fcaf:~/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap# make
cc -pthread -O3 -Wall -g -I../phc-winner-argon2/include -I../phc-winner-argon2/src -shared -fPIC ../phc-winner-argon2/src/argon2.c ../phc-winner-argon2/src/core.c ../phc-winner-argon2/src/blake2/blake2b.c ../phc-winner-argon2/src/thread.c ../phc-winner-argon2/src/encoding.c argon_wrap.c ../phc-winner-argon2/src/ref.c -o libargon2_wrap.so
root@7e9fcf33fcaf:~/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/argon2-2.1.3/ext/argon2_wrap# cd
root@7e9fcf33fcaf:~# irb
irb(main):001:0> require 'argon2'
=> true
irb(main):002:0>

communiteq · 2022-12-26T12:33:48Z

Hmm why is extconf.rb empty?

4e70673#diff-cbcaa657275e9aedc9400e59a95de0559a89551fd82ef78ce5bc1aaf99f6276d

technion · 2022-12-26T12:45:18Z

@communiteq Because I had Makefile that took care of the external libraries and didn't need to learn another Ruby DSL to make it build one for me. They may well have broken that.

technion · 2022-12-26T13:42:25Z

It appears that is in fact what's going on here, it simply doesn't run the build. I've started with mkmf and it definitely tries to build then, but there's a rabbit hole of not finding the correct files and so on.

davekaro · 2022-12-26T13:47:52Z

it simply seems to never be running the actual build script.

yeah I think this is what I was seeing too, based on the missing files in the ext/argon2_wrap. The reason it works when building the gem locally and then adding lib to the $LOAD_PATH was because the locally built gem had the files. I don't think I'm saying anything you don't already know at this point. It is weird though because there is the output when installing the gem:

Fetching argon2-2.1.3.gem
Building native extensions. This could take a while...

and it takes the same amount of time as installing on 3.1...... so it does seem it's running the build script and doing something.

technion · 2022-12-26T13:51:22Z

I need to take a break as it's 1am here, but I'm working on implementing the build with:

https://ruby-doc.org/stdlib-2.5.1/libdoc/mkmf/rdoc/MakeMakefile.html

Currently it definitely builds, but half the code is missing and functions fail. I'm struggling to work this tooling, which again, is why I originally used an existing Makefile. I have no idea why that stopped being valid.

scott-knight · 2022-12-26T15:47:48Z

Done. Can anyone else check with version 2.1.3 and confirm?

@technion, last night, after downgrading back to Ruby 3.1.3, I had the same issue as Ruby 3.2. I haven't been able to pinpoint what caused it. So, I'm in the process of creating a clean install of my local build. Once I have it running, I will let you know what I find.

davekaro · 2022-12-26T16:33:36Z

@scott-knight I’m not by my computer but curious if when you downgraded ruby whether rubygems also downgraded or stayed the same? I’m wondering if the issue is in a new rubygems release that may help narrow down what is wrong.

j-manu · 2022-12-26T17:25:37Z

I'm facing the same problem with a different gem with Ruby 3.1.2.

I have made a minimal reproduction here

https://github.com/debug-poc/cld3

https://github.com/debug-poc/cld3/actions/runs/3781540490/jobs/6428505336

I think it's the rubygems update that's causing the problem. If you change the Dockerfile above to remove the rubygems update it should work.

scott-knight · 2022-12-26T17:39:08Z

@scott-knight I’m not by my computer but curious if when you downgraded ruby whether rubygems also downgraded or stayed the same? I’m wondering if the issue is in a new rubygems release that may help narrow down what is wrong.

@davekaro That's a great question. I was in a hurry last night (was heading out to a Christmas party) as I was trying to get things returned to the previous state. If I recall correctly, I actually deleted the entire gem dir and rebundled under Ruby 3.1.3. This morning, not recalling everything that actually happened, and believing my distro was corrupt, I just dropped the distro.

Now, after reading what @j-manu just posted (and then re-reading your question), that actually makes sense that rubygem-update is the cause. Once I downgraded to 3.1.3, I ran gem update --system to update my local install of rubygem-update and updated to bundler 2.4.1... (it's all coming back to me now).

scott-knight · 2022-12-26T20:38:07Z

@technion I just wrapped up creating my new distro -- Ubuntu 23.04, Ruby 3.2.0, Rails 7.0.4. The gems all installed perfectly without issue. Additionally, I was able to bundle update and the new argon2 2.1.3 updated. However, when I tried to run my test suite, I still get the same error:

When I try to run the rails server:

/home/sknight/proj-dir/app-name/.gems/gems/ffi-compiler-1.0.1/lib/ffi-compiler/loader.rb:21:in `find': cannot find 'argon2_wrap' library (LoadError)
        from /home/sknight/proj-dir/app-name/.gems/gems/argon2-2.1.3/lib/argon2/ffi_engine.rb:10:in `<module:Ext>'
        from /home/sknight/proj-dir/app-name/.gems/gems/argon2-2.1.3/lib/argon2/ffi_engine.rb:8:in `<module:Argon2>'
        from /home/sknight/proj-dir/app-name/.gems/gems/argon2-2.1.3/lib/argon2/ffi_engine.rb:6:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/argon2-2.1.3/lib/argon2.rb:4:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/devise-argon2-1.1.0/lib/devise/encryptable/encryptors/argon2.rb:1:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/devise-argon2-1.1.0/lib/devise-argon2.rb:4:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler/runtime.rb:60:in `block (2 levels) in require'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler/runtime.rb:55:in `each'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler/runtime.rb:55:in `block in require'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler/runtime.rb:44:in `each'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler/runtime.rb:44:in `require'
        from /home/sknight/.rbenv/versions/3.2.0/lib/ruby/gems/3.2.0/gems/bundler-2.4.1/lib/bundler.rb:195:in `require'
        from /home/sknight/proj-dir/app-name/config/application.rb:19:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:137:in `block in perform'
        from <internal:kernel>:90:in `tap'
        from /home/sknight/proj-dir/app-name/.gems/gems/railties-7.0.4/lib/rails/commands/server/server_command.rb:134:in `perform'
        from /home/sknight/proj-dir/app-name/.gems/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /home/sknight/proj-dir/app-name/.gems/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /home/sknight/proj-dir/app-name/.gems/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /home/sknight/proj-dir/app-name/.gems/gems/railties-7.0.4/lib/rails/command/base.rb:87:in `perform'
        from /home/sknight/proj-dir/app-name/.gems/gems/railties-7.0.4/lib/rails/command.rb:48:in `invoke'
        from /home/sknight/proj-dir/app-name/.gems/gems/railties-7.0.4/lib/rails/commands.rb:18:in `<main>'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'
        from /home/sknight/proj-dir/app-name/.gems/gems/bootsnap-1.15.0/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:32:in `require'

#############################################################################

DOWNGRADING TO RUBY 3.1.3:

With Ubuntu 23.04, Ruby 3.1.3, Rails 7.0.4. I was also able to bundle update to the new argon2 2.1.3 gem. Low-and-behold, everything works as expected! rspec runs and the rails server runs.

However, to get this to work, I had to delete my local .gems dir which was bundled via Ruby 3.2.0. Once removed, I re-bundled with Ruby 3.1.3, without updating rubygems-update to 3.4.0, and everything worked as expected.

On a side note: before re-bundling with Ruby 3.1.3, when I tried to run rspec with the Ruby 3.2.0 bundled gems, this happened:

Ignoring argon2-2.1.3 because its extensions are not built. Try: gem pristine argon2 --version 2.1.3
Ignoring bcrypt-3.1.18 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.18
Ignoring bindex-0.8.1 because its extensions are not built. Try: gem pristine bindex --version 0.8.1
Ignoring bootsnap-1.15.0 because its extensions are not built. Try: gem pristine bootsnap --version 1.15.0
Ignoring ffi-1.15.5 because its extensions are not built. Try: gem pristine ffi --version 1.15.5
Ignoring haml-6.1.1 because its extensions are not built. Try: gem pristine haml --version 6.1.1
Ignoring hamlit-3.0.3 because its extensions are not built. Try: gem pristine hamlit --version 3.0.3
Ignoring msgpack-1.6.0 because its extensions are not built. Try: gem pristine msgpack --version 1.6.0
Ignoring nio4r-2.5.8 because its extensions are not built. Try: gem pristine nio4r --version 2.5.8
Ignoring nokogiri-1.13.10 because its extensions are not built. Try: gem pristine nokogiri --version 1.13.10
Ignoring oj-3.13.23 because its extensions are not built. Try: gem pristine oj --version 3.13.23
Ignoring pg-1.4.5 because its extensions are not built. Try: gem pristine pg --version 1.4.5
Ignoring puma-6.0.1 because its extensions are not built. Try: gem pristine puma --version 6.0.1
Ignoring websocket-driver-0.7.5 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.5
Could not find pg-1.4.5, puma-6.0.1, bootsnap-1.15.0, hamlit-3.0.3, oj-3.13.23, debug-1.7.1, nio4r-2.5.8, msgpack-1.6.0, ffi-1.15.5, bcrypt-3.1.18, argon2-2.1.3, irb-1.6.2, reline-0.3.2, haml-6.1.1, nokogiri-1.13.10-x86_64-linux, bindex-0.8.1, websocket-driver-0.7.5, net-imap-0.3.4, net-pop-0.1.2, net-smtp-0.3.3, minitest-5.16.3, io-console-0.6.0, racc-1.6.2, json-2.6.3, date-3.3.3, net-protocol-0.2.1, timeout-0.3.1 in locally installed gems
Run `bundle install` to install missing gems.
Ignoring argon2-2.1.3 because its extensions are not built. Try: gem pristine argon2 --version 2.1.3
Ignoring bcrypt-3.1.18 because its extensions are not built. Try: gem pristine bcrypt --version 3.1.18
Ignoring bindex-0.8.1 because its extensions are not built. Try: gem pristine bindex --version 0.8.1
Ignoring bootsnap-1.15.0 because its extensions are not built. Try: gem pristine bootsnap --version 1.15.0
Ignoring ffi-1.15.5 because its extensions are not built. Try: gem pristine ffi --version 1.15.5
Ignoring haml-6.1.1 because its extensions are not built. Try: gem pristine haml --version 6.1.1
Ignoring hamlit-3.0.3 because its extensions are not built. Try: gem pristine hamlit --version 3.0.3
Ignoring msgpack-1.6.0 because its extensions are not built. Try: gem pristine msgpack --version 1.6.0
Ignoring nio4r-2.5.8 because its extensions are not built. Try: gem pristine nio4r --version 2.5.8
Ignoring nokogiri-1.13.10 because its extensions are not built. Try: gem pristine nokogiri --version 1.13.10
Ignoring oj-3.13.23 because its extensions are not built. Try: gem pristine oj --version 3.13.23
Ignoring pg-1.4.5 because its extensions are not built. Try: gem pristine pg --version 1.4.5
Ignoring puma-6.0.1 because its extensions are not built. Try: gem pristine puma --version 6.0.1
Ignoring websocket-driver-0.7.5 because its extensions are not built. Try: gem pristine websocket-driver --version 0.7.5

While loading ./spec/models/user_spec.rb an `exit` / `raise SystemExit` occurred, RSpec will now quit.
Failure/Error: require File.expand_path('../config/environment', __dir__)

SystemExit:
  exit

It appears there are several gems that may have this issue with the new 3.4.0 rubygems-update

I wish I could be more help.

technion · 2022-12-26T20:56:17Z

Thanks for the extra info @scott-knight . If it's caused by a rubygems update that may explain why none of my existing systems can replicate this issue, but some people here can.

In summary, simply having a Makefile in the right folder it used to just run it. I really don't get why the ecosystem preferred MakeMakefile, which is proving frustratingly complicated when you have more than the basic case.

scott-knight · 2022-12-26T21:39:40Z

@technion, I was just reviewing at the 3.4.0 rubygems-update, looking at the new resolver code. In the resolver code, they're force deleting dependencies where dependencies are out of scope for the current platform (at least that's how comment reads). This also looks like the only place in their code where this might be causing an issue (since they're actively deleting packages). I believe the effects of this new behavior can be seen in my post above where a number of gems report their extensions are not built. How a resolution occurs to install a needed dependency (after one has been deleted) isn't apparent from this code update.

How this would help you fix your gem, I don't know. But I hope this helps somehow. ( If not... sorry. I tried :-) )

communiteq · 2022-12-26T22:12:53Z

I have tested some more and I see that this broke between rubygems-update version 3.3.26 and 3.4.0. The first one works, the latter one breaks.

technion · 2022-12-26T23:04:22Z

Thanks for your patience and help everyone.

I've just pushed a rewrite of the Makefile and it appears to work.

root@7ffc7d633108:/ruby-argon2# ruby -v
ruby 3.2.0 (2022-12-25 revision a528908271) [x86_64-linux]
root@7ffc7d633108:/ruby-argon2# gem install argon2-2.1.3.gem
Building native extensions. This could take a while...
Successfully installed argon2-2.1.3
Parsing documentation for argon2-2.1.3
Installing ri documentation for argon2-2.1.3
Done installing documentation for argon2 after 2 seconds
1 gem installed

irb(main):001:0> require 'argon2'
=> true
irb(main):002:0> hasher = Argon2::Password.new
irb(main):003:0> hasher.create("password")
=> "$argon2id$v=19$m=65536,t=2,p=1$yt9iCVsmXYkWmkJ4YqhYaw$/74dvohSpLR/ix7iUG0M1EugxazeYm2y5OKPg2luY5Y"

It does however.. cause the test suite to fail. Working on that next.

hsbt · 2022-12-27T02:52:36Z

Hi, I'm rubygems/bundler maintainer.

It seems to relate rubygems/rubygems#6133. argon2-2.1.2/ext/argon2_wrap/libargon2_wrap.bundle is removed after gem installation with RubyGems 3.4.x bundled with Ruby 3.2.

I wonder why ffi use build artifact from build directory, not installation directory.

technion · 2022-12-27T03:07:42Z

@hsbt Many thanks for chiming in here. In the latest commit I've finished moving things to the more "common" method of using MakeMakefile and hopefully we have an environment more like you're used to, and I'm convinced there's a bug involving not using CFLAGS. If I hack my way around the below issue, we have working code. From my extconf.rb:

 append_cflags " -fPIC -shared"

If I run "ruby extconf.rb" and look at the created Makefile, it clearly shows CFLAGS setting those options. However if I enable V=1 and run make, it ignores those flags and then fails to link accordingly.

x86_64-linux-gnu-gcc -I. -I/usr/include/x86_64-linux-gnu/ruby-3.0.0 -I/usr/include/ruby-3.0.0/ruby/backward -I/usr/include/ruby-3.0.0 -I. -I../phc-winner-argon2/include -I../phc-winner-argon2/src -Wdate-time -D_FORTIFY_SOURCE=2   -pthread -O3 -Wall -g -I../phc-winner-argon2/include -I../phc-winner-argon2/src -o ../phc-winner-argon2/src/argon2.o -c ../phc-winner-argon2/src/argon2.c
/usr/bin/ld: ../phc-winner-argon2/src/core.o: warning: relocation against `FLAG_clear_internal_memory' in read-only section `.text'
/usr/bin/ld: ../phc-winner-argon2/src/core.o: relocation R_X86_64_PC32 against symbol `FLAG_clear_internal_memory' can not be used when making a shared object; recompile with -fPIC
...

As for ffi's file locations, I don't know but that isn't the root issue, the file isn't being created at all in my original build. I think it's as simple as saying, I used to just have a blank extconf.rb, and it would be assumed the Makefile there was valid. Now, without a create_makefile() command it assumes there isn't one and never builds it.

hsbt · 2022-12-27T03:24:34Z

@technion This issue may caused by https://github.com/ffi/ffi-compiler/blob/master/lib/ffi-compiler/loader.rb#L12 logic. But I confirmed another issue of ox.

technion · 2022-12-27T03:35:56Z

Commit 927bf0e reverts everything back to my original Makefile, it came to my attention there was even more platform dependent code that was going to need porting and I was slowly losing my mind over it.

I've introduced a horrible workaround to this whole issue, in the form of adding this line to extconf.rb:

system "make"

I hate it, but here we are, the CI works. I've created a package because testing this gem from github has its own challenges with the submodule.

root@780092a29cb2:/# ruby -v
ruby 3.2.0rc1 (2022-12-22) [x86_64-linux]
root@780092a29cb2:/# wget https://ctadvisor.lolware.net//argon2-2.2.0.gem
root@780092a29cb2:/# gem install argon2-2.2.0.gem
Building native extensions. This could take a while...
Successfully installed argon2-2.2.0
1 gem installed
root@780092a29cb2:/# irb
irb(main):001:0> require 'argon2'
=> true
irb(main):002:0> hasher = Argon2::Password.new(t_cost: 2, m_cost: 16, p_cost: 1)
irb(main):003:0> hasher.create("password")
=> "$argon2id$v=19$m=65536,t=2,p=1$CP2ii3RBHWecd+XhNOcyNw$Lka/44bkEuaGX2/8FFR+7axu4tDcph8jrvrX+ACXwJ0"
irb(main):004:0>

If anyone would like to test the above package it would be appreciated. I'm just cleaning the Rubocop errors and should be able to tag a release soon.

I think this confirms the issue is not ffi-loader's resolution, it's when you type "gem install" it simply never runs "make" like it should.

hsbt · 2022-12-27T03:45:48Z

Thanks for explanation. I didn't investigate ffi-compiler yet. It's great to fix only ruby-argon2 side.

If you found another loading path or build artifact issues, please ping me again.

yob · 2022-12-27T03:49:53Z

@hsbt if you're interested in another example, I've noticed the same symptoms with ruby 3.2.0 and rubygems 3.4 in https://github.com/dbalatero/levenshtein-ffi. The gem hasn't seen a release in > 8 years, but it was working ok on ruby 3.1.x

technion · 2022-12-27T05:33:35Z

@hsbt I've come up with a substantively better workaround, removed the "system" call and just added a cp to my Makefile to move the library something your new Rubygems doesn't clean up. If anyone would like to try the updated https://ctadvisor.lolware.net//argon2-2.2.0.gem it would be appreciated, I'm hoping we have a proper solution finally.

communiteq · 2022-12-27T07:45:02Z

In Discourse, I can only test when it's actually on Rubygems, so I am of no use here right now.

In the meanwhile I propose to change the title of this issue to "Fails to load with Rubygems 3.4", for future reference.

technion · 2022-12-27T08:29:06Z

Version 2.2.0 has just been pushed. Let's cross fingers this is behind us.

communiteq · 2022-12-27T09:26:34Z

For me 2.2.0 resolves the issue!!!! Thank you @technion ❤️

hsbt · 2022-12-27T10:41:54Z

👍 Thanks all.

hsbt mentioned this issue Dec 27, 2022

Cleaning build artifact broke some of gems rubygems/rubygems#6205

Closed

dkam mentioned this issue Dec 27, 2022

Ruby 3.2 compatibility ohler55/ox#300

Closed

technion added a commit that referenced this issue Dec 27, 2022

Bump version. New horrible solution to #56

27793ab

communiteq added a commit to communiteq/discourse-migratepassword that referenced this issue Dec 27, 2022

Fix for RubyGems 3.4 technion/ruby-argon2#56

f787742

technion closed this as completed Dec 27, 2022

davekaro changed the title ~~Fails to load on Ruby 3.2~~ Fails to load with Rubygems 3.4 Dec 27, 2022

schorsch mentioned this issue Mar 8, 2023

update argon2 to v2.2.0 due to errors loading argon2_wrap library thoughtbot/clearance#989

Merged

Fails to load with Rubygems 3.4 #56

Fails to load with Rubygems 3.4 #56

Comments

davekaro commented Dec 25, 2022 • edited Loading

scott-knight commented Dec 25, 2022 • edited Loading

communiteq commented Dec 25, 2022

technion commented Dec 26, 2022

technion commented Dec 26, 2022

technion commented Dec 26, 2022

technion commented Dec 26, 2022

communiteq commented Dec 26, 2022

technion commented Dec 26, 2022

communiteq commented Dec 26, 2022

technion commented Dec 26, 2022

technion commented Dec 26, 2022

communiteq commented Dec 26, 2022

technion commented Dec 26, 2022

technion commented Dec 26, 2022

davekaro commented Dec 26, 2022

technion commented Dec 26, 2022

scott-knight commented Dec 26, 2022 • edited Loading

davekaro commented Dec 26, 2022

j-manu commented Dec 26, 2022

scott-knight commented Dec 26, 2022 • edited Loading

scott-knight commented Dec 26, 2022 • edited Loading

technion commented Dec 26, 2022

scott-knight commented Dec 26, 2022 • edited Loading

communiteq commented Dec 26, 2022 • edited Loading

technion commented Dec 26, 2022

hsbt commented Dec 27, 2022 • edited Loading

technion commented Dec 27, 2022

hsbt commented Dec 27, 2022

technion commented Dec 27, 2022

hsbt commented Dec 27, 2022

yob commented Dec 27, 2022

technion commented Dec 27, 2022

communiteq commented Dec 27, 2022

technion commented Dec 27, 2022

communiteq commented Dec 27, 2022

hsbt commented Dec 27, 2022

davekaro commented Dec 25, 2022 •

edited

Loading

scott-knight commented Dec 25, 2022 •

edited

Loading

scott-knight commented Dec 26, 2022 •

edited

Loading

scott-knight commented Dec 26, 2022 •

edited

Loading

scott-knight commented Dec 26, 2022 •

edited

Loading

scott-knight commented Dec 26, 2022 •

edited

Loading

communiteq commented Dec 26, 2022 •

edited

Loading

hsbt commented Dec 27, 2022 •

edited

Loading