Fix gem installations that rely on OpenSSL when build_package_mac_openssl() is used

[pstengel]

This PR is an alternate approach to #1439. Credit goes to @puyo for identifying the issue.

Looking through the code, I believe using package_option is the preferred way to add configuration options. I took @puyo's fix and converted it to use package_option instead.

I can confirm this fix works—I was having the same issue compiling mysql2 on macOS Catalina. It's no longer a problem with this change in place.

[puyo]

Thank you! I suspected I wasn't doing things in a way that was correct or consistent with the rest of the repo but I wasn't sure what the right way was. I'll be very happy when this is fixed, I'll remove a diff+patch step from my company's tool chain.

[puyo]

Bump! I would find it handy if this were merged.

[hsbt]

--with-opt-dir is not only affect with OpenSSL. Readline, Zlib, and other libraries relied on --with-opt-dir.

mysql2 issue was solved with brianmario/mysql2#1005 (comment) . We should use the same ssl library with mysql package. So, We shouldn't link the other installed library like openssl built by ruby-build.

[puyo]

@hsbt Do you want such gems to install transparently and successfully using the version of openssl that is downloaded and used for the built-in Ruby libraries, or do you want to have to look up the bundle install command you refer to either on Stackoverflow or here, and run that in order to successfully configure and install such gems?

[coding-red-panda]

Is there any update regarding this approach? Because with the recent updates from Command Line Tools etc, it's becoming harder to installer old Ruby versions on a Macbook. Downgrading the Command Line Tools allows us to build Ruby regarding older openSSL by linking it through Homebrew (or relying on the Ruby build one), but then gems like mysql stop compiling, and the linked solution does not solve it.

[puyo]

Yes, I think the reason openssl is downloaded by ruby-build in the first place is because of problems with using a brew'd version on a newer Mac or other OS.

[coding-red-panda]

Yes, I think the reason openssl is downloaded by ruby-build in the first place is because of problems with using a brew'd version on a newer Mac or other OS.

Yes and it still does on the Catalina with the final version. But on the latest command line tools (clang 12) it seems that whatever ruby build does it’s ignored and the compilation keeps failing since none of the required libs such as OpenSSL, readline etc are compiled

[hsbt]

We shouldn't care mysql2 issues in ruby-build. Unfortunately, you can resolve this issue with brianmario/mysql2#1005 (comment)

[puyo]

We shouldn't care mysql2 issues in ruby-build

• mysql2 is just an example.
• This problem affects all gems that are compiled against libopenssl.
• Nothing in the fix refers to a specific gem such as mysql2.
• I now have about 10 engineers using the patch in this PR full time and reporting no problems.

[JulianLeviston]

@hsbt 失礼しますが、このPRはいいと思います。Excuse us, but we think this PR is still good.

Please reopen this PR. 🙏 🥺

The code already has: package_option ruby configure --with-openssl-dir="$OPENSSL_PREFIX_PATH".
We also need package_option ruby configure --with-opt-dir="$OPENSSL_PREFIX_PATH" so installations with OPENSSL don't break when installing with asdf.

🙏 🥺

どうもありがとうございます。Thankyou very much.

[puyo]

This is still a problem, this PR still fixes the problem.

Without this PR:

brew install openssl
gem install $ANY_GEM_THAT_REQUIRES_OPENSSL -- --with-opt-dir="$(brew --prefix openssl)"

With this PR:

gem install $ANY_GEM_THAT_REQUIRES_OPENSSL

[mislav]

If a Ruby gem with native extensions needs to link to OpenSSL and if OpenSSL is in a non-standard location (such as the one provided by Homebrew), then the correct solution is to explicitly provide that location when installing the gem in question.

# correct solution
gem install mysql2 -- --with-opt-dir="$(brew --prefix openssl)"

This is not a bug.

[puyo]

If a Ruby gem with native extensions needs to link to OpenSSL and if OpenSSL is in a non-standard location (such as the one provided by Homebrew), then the correct solution is to explicitly provide that location when installing the gem in question.

This is not a bug.

I want to clarify, since I think a reader of this PR would get the wrong idea from this statement.

The statement is true, gems that rely on a specific version of OpenSSL such as that provided by Homebrew should use the solution you suggest (and a different solution if they're using Bundler). -> brianmario/mysql2#1005 (comment)

But that is not the problem we are trying to solve with this PR, and that is why I need to clarify.

Every gem we have that uses openssl, across all of our organisation's repositories and gems, works fine with the openssl that is downloaded and used by ruby-build. So, this PR avoids anybody's need to brew install any openssl packages, or use any of the solutions cited. This PR makes gems that use openssl "just work" without any further configuration or work, whether using gem install or including such gems in a Gemfile. For us developers, it makes ruby-build work the way we expect it to. It may not be a bug, it may be a feature of ruby-build that you must use customised gem and bundle commands, but I would characterise it as an inconvenience.

We've been using this PR for years now. Before this PR, we had new developers and para-development staff such as QA stumbling whenever they ran bundle on projects that had mysql2 in them. They weren't in a position to debug and fix this with custom gem commands, they were just running the scripts that developers gave them. We updated our scripts to include this patch in order to make everything work for them. I was only hoping this PR would offer a chance for us to delete the patching that goes on downstream and simplify our scripts, and might be of benefit to other users of ruby-build.

[pstengel]

I figured I'd chime in with my viewpoint since I ran across this issue at least once, and it bugged me enough to open a PR for it.

I view tools like ruby-build as "easy buttons." They're supposed to take something that we have to do daily that was a complete pain in the ass and make it trivial. So if I had to come up with ruby-build's mission statement, it would be to install Ruby versions that "just work" on the platforms the project chooses to support.

Since ruby-build is installing Ruby, it's also installing the gem tool. If I install a version of Ruby that reproduces this issue and run gem install mysql2, it doesn't work. That's not making things that much easier on me. Sure, it installed Ruby, but I can't use one of Ruby's standard features without additional work.

To me, that's a bit like taking delivery of a new car with one of the doors missing. Is it a car? Yes. Does it run and get me places? Sure. Is it the year, model, and color I ordered? Yup. But it's missing a damn door! And that's inconvenient and certainly not what I expected. Predictable, consistent expectations are everything in software, and people expect gem installs to succeed, particularly when they use standard libraries already required for Ruby's compilation.

That said, I wonder if the middle ground here is less about ensuring OpenSSL itself works. Perhaps we should focus more on ensuring that any gem we ask to install that uses a standard library that Ruby already compiles against can find that same library as ruby-build provides it.

For example, readline is another library that ruby-build will automatically manage as required. I'm pretty sure that if I had a version of Ruby installed that required ruby-build to supply readline, and I tried to install a gem that relied on readline, I would get a similar error to what we get now with gems that depend on OpenSSL.

So if we moved package_option ruby configure --with-opt-dir="$OPENSSL_PREFIX_PATH" to, say, build_package_standard_build() and changed the flag to --with-opt-dir="$PREFIX_PATH" to solve the problem for all bundled libraries, would the ruby-build team be more open to considering it for addition?

(Granted, I haven't tested that, and I'm not 100% sure it would work, but my question remains.)

[puyo]

It's a good point about readline.

[mislav]

Thanks for explaining. Now that I read this further, I see that the original motivation for this PR wasn't to necessarily make the brew install [email protected] case better, but to make it better when we automatically build a Ruby-specific openssl version on Mac, to ensure that the gems are also automatically linked to that version. Since the user never manually chose to set up the openssl version in this case, but ruby-build did transparently behind the scenes, then I think the change makes sense, since we can't expect the user to link their gems to an openssl version that they don't know even where it is.

So if we moved package_option ruby configure --with-opt-dir="$OPENSSL_PREFIX_PATH" to, say, build_package_standard_build() and changed the flag to --with-opt-dir="$PREFIX_PATH" to solve the problem for all bundled libraries, would the ruby-build team be more open to considering it for addition?

I would certainly be more open to that. @hsbt How would you feel about this compromise?

[hsbt]

In my investigation, @puyo and @pstengel are miss-understanding. mysql2 is NOT depends on OpenSSL. So, this root cause is that libmysqlclient build with OpenSSL provided by homebrew. Like this.

otool -L /opt/homebrew/lib/libmysqlclient.dylib:
        /opt/homebrew/opt/mysql/lib/libmysqlclient.21.dylib (compatibility version 21.0.0, current version 21.0.0)
        /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
        /opt/homebrew/opt/zstd/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.2)
        /opt/homebrew/opt/[email protected]/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
        /opt/homebrew/opt/[email protected]/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0)
        /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

I used over the 400+ gems for my work. But this problem only happened mysql2 gem because this issue libmysqlclient depends on homebrew package, not gem.

In the other hand, this issue already fixed by upstream at brianmario/mysql2@e2503dc. I still believe this issue should be fixed mysql2 gem side.

[pstengel]

I could be misunderstanding—it's been a while since I dealt with this issue. However, if I have time later today, I'll see if I can put together a proof for discussion here.

[mislav]

Please do investigate and report back. If this issue only affects the mysql2 gem, and the fix is already in mysql2 upstream, then I'm on @hsbt side and I would rather not make any change to how Ruby is built.

[puyo]

Please do investigate and report back.

Today I removed my downstream patch and reinstalled all my asdf ruby installations from the ground up. I'll let you know if I encounter any openssl problems with any gems. If was just mysql2 and now that works, then I won't worry about other potential problems with other gems.

[hsbt]

mysql2-0.5.4 fixed this issue.

https://github.com/brianmario/mysql2/releases/tag/0.5.4