SEGV Fault error when starting worker

[jmarsh24]

I'm running into the following SEGV error when trying to boot up the solid_queue worker. I have suspicions that it's related to the mac os version because a colleague of mine has an m2 on 13.6.3 and it boots fine. I'm not sure if this is the place to post it, but this error only comes up with this gem so far. I haven't had any other issues.

rails version 7.1.3
ruby version 3.1.4 (rbenv)
m2 macbook pro
Sonoma 14.2.1

bundle exec rake solid_queue:start
DEPRECATION WARNING: DeprecatedConstantAccessor.deprecate_constant without a deprecator is deprecated (called from require at <internal:/Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37)
warning: parser/current is loading parser/ruby32, which recognizes 3.2.3-compliant syntax, but you are running 3.2.2.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.
[SolidQueue] Starting Dispatcher(pid=10816, hostname=Justins-MacBook-Pro.local, metadata={:polling_interval=>1, :batch_size=>500})
[SolidQueue] Starting Worker(pid=10817, hostname=Justins-MacBook-Pro.local, metadata={:polling_interval=>0.1, :queues=>"*", :thread_pool_size=>5})
/Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:690: [BUG] Segmentation fault at 0x0000000105ef0ace
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin23]

-- Crash Report log information --------------------------------------------
   See Crash Report log file in one of the following locations:
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.
Don't forget to include the above Crash Report log file in bug reports.

-- Control frame information -----------------------------------------------
c:0116 p:---- s:0700 e:000699 CFUNC  :connect_poll
c:0115 p:0358 s:0696 e:000695 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:690
c:0114 p:0265 s:0683 e:000682 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:824
c:0113 p:0007 s:0672 e:000671 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:759
c:0112 p:0012 s:0666 e:000665 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg.rb:63
c:0111 p:0006 s:0660 e:000659 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/postgresq
c:0110 p:0008 s:0654 SEGV received in SEGV handler
e:000653 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/postgresq
c:0109 p:0020 s:0649 e:000648 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/postgresq
c:0108 p:0004 s:0645 e:000644 BLOCK  /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activerecord-7.1.3/lib/active_record/connection_adapters/abstract_
c:0107 p:0002 s:0639 e:000638 METHOD /Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/activesupport-7.1.3/lib/active_support/concurrency/null_lock.rb:9
c:0106 p:0031 [SolidQueue] Restarting fork[10816] (status: )
[SolidQueue] Starting Dispatcher(pid=10824, hostname=Justins-MacBook-Pro.local, metadata={:polling_interval=>1, :batch_size=>500})
/Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:690: [BUG] Segmentation fault at 0x0000000105ef0ace
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin23]

-- Crash Report log information --------------------------------------------
   See Crash Report log file in one of the following locations:
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports                           SEGV received in SEGV handler

   for more details.                 [SolidQueue] Restarting fork[10824] (status: )
[SolidQueue] Starting Dispatcher(pid=10825, hostname=Justins-MacBook-Pro.local, metadata={:polling_interval=>1, :batch_size=>500})
/Users/jm_mbp/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/gems/pg-1.5.4/lib/pg/connection.rb:690: [BUG] Segmentation fault at 0x0000000105ef0ace
ruby 3.2.2 (2023-03-30 revision e51014f9c0) [arm64-darwin23]

-- Crash Report log information --------------------------------------------
   See Crash Report log file in one of the following locations:
     * ~/Library/Logs/DiagnosticReports
     * /Library/Logs/DiagnosticReports
   for more details.
Don't forget to include the above Crash Report log file in bug reports.

[rosa]

Huh, thanks for reporting @jmarsh24. I see these two lines:

warning: parser/current is loading parser/ruby32, which recognizes 3.2.3-compliant syntax, but you are running 3.2.2.
Please see https://github.com/whitequark/parser#compatibility-with-ruby-mri.

Are you running https://github.com/whitequark/parser? could you try switching to Ruby MRI and see if the segfault still happens?

[jmarsh24]

@rosa I'm not using a custom interpreter, I think it's a dependent of rubocop. I use rbenv install 3.2.3 and it seems to have removed the warning. I also tried to boot solid_queue as a plugin but still ran into the error. I have included both logs below.

bin/dev
system | Tmux socket name: overmind-app-VcAKauTRD2_8xqcJy0B9C
system | Tmux session ID: app
system | Listening at ./.overmind.sock
web    | Started with pid 31252...
js     | Started with pid 31253...
yarn run v1.22.21
$ esbuild app/javascript/*.* --bundle --sourcemap --outdir=app/assets/builds --watch
js     | [watch] build finished, watching for changes...
web    | DEPRECATION WARNING: DeprecatedConstantAccessor.deprecate_constant without a deprecator is deprecated (called from require at <internal:/Users/jm_mbp/.rbenv/versions/3.1.4/lib/ruby/site_ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:37)
web    | => Booting Puma
web    | => Rails 7.1.3 application starting in development
web    | => Run `bin/rails server --help` for more startup options
web    | Puma starting in single mode...
web    | * Puma version: 6.3.0 (ruby 3.1.4-p223) ("Mugi No Toki Itaru")
web    | *  Min threads: 20
web    | *  Max threads: 20
web    | *  Environment: development
web    | *          PID: 31274
web    | * Listening on http://127.0.0.1:3000
web    | * Listening on http://[::1]:3000
web    | Use Ctrl-C to stop
web    | objc[31312]: +[NSCharacterSet initialize] may have been in progress in another thread when fork() was called.
web    | objc[31312]: +[NSCharacterSet initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
web    | Detected Solid Queue has gone away, stopping Puma...
web    | - Gracefully stopping, waiting for requests to finish
web    | === puma shutdown: 2024-02-06 13:49:00 +0100 ===
web    | - Goodbye!
web    | Exiting
web    | Shutting down background worker
web    | Killing session flusher
web    | Exited
js     | Interrupting...
js     | Exited

This is the SEGV fault log after upgrading
https://gist.github.com/jmarsh24/470490d0bb548a6cb0e41ec33026df0a

[jmarsh24]

@rosa False alarm! I think I fixed the issue. The pg config was pointing to an old path installed by the postgres app instead of home-brew. I cleared up the config to point to the correct path and now it boots up without a memory error.

[rosa]

Ah, great! Thanks for letting me know 🙏

[JoshReedSchramm]

I know this is closed but I'm seeing a very similar behavior and its not the postgres thing. It's intermittent - if i restart the runner it will come up just fine but periodically seg faults. It is not happening on the ubuntu based servers - only macos and i think it's related to the m1+ chipset - none of the devs on the team with older macs have reported the issue.

Stack track is basically identical to the original report except i use RVM instead of rbenv.

I'm not great at debugging these kinds of failures but I'm happy to work on it with anyone if other folks are seeing it.

[jmarsh24]

@JoshReedSchramm I had messed up my Postgres installation at some point which is what was causing this error. Check the path that is being used and verify that it's to the current Postgres installation. Homebrew, Postgres.app, or any other installation and see if that fixes it.

At one point you could have been on Rosetta and upgraded to arm64 but didn't properly reinstall it.

[SjoerdL]

@JoshReedSchramm I had the same thing and for me it was a problem with postgres of MacOS, namely this issue: ged/ruby-pg#538. Setting export PGGSSENCMODE="disable" fixed it for me, as suggested here: ged/ruby-pg#538 (comment)

[adrianthedev]

This helped me as well. Thanks!

@JoshReedSchramm I had the same thing and for me it was a problem with postgres of MacOS, namely this issue: ged/ruby-pg#538. Setting export PGGSSENCMODE="disable" fixed it for me, as suggested here: ged/ruby-pg#538 (comment)

[feapaydin]

Regarding this issue, a detailed explanation and alternative solutions are listed here: ged/ruby-pg#311 (comment)

Especially if you're running postgres inside a docker container like myself, alternatives listed by stanhu can be useful:

• Disable GSSAPI via the PGGSSENCMODE=disable environment variable or pass gssencmode=disable in the connection string.
• Initiate the database connection by preloading the app before forking so the macOS system calls are invoked in the parent process. In Puma, the preload_app! config option does this.
• Use a PostgreSQL server that does not have --with-gssapi enabled. By default I believe the Homebrew version has this, but asdf does not install PostgreSQL with this. You can check by using otool -L: