Nokogiri Build Failure: Bad Design

[wjk]

Whenever I try to compile Nokogiri for OS X, I get a failure message because /usr/include/iconv.h is not found on my computer. This is because Nokogiri is hardcoding that exact path into its extconf.rb script. Even though iconv.h is very much present (inside my Xcode SDK), and even though Nokogiri compiles just fine when I manually patch extconf.rb to remove the check, the fact is that Nokogiri won’t play nice.

While I can manually patch Nokogiri to remove this shortsighted check, the fact is that Nokogiri is required in so many places that manually patching it every time is infeasible or impossible. Sadly, this check has been in place for so long that, even if you agree with me and decide to fix it, it won’t help me much because I have gems that depend on versions that have already been pushed that include the check, and that cannot be changed for stability reasons. Does anyone have any ideas as to how this could be resolved?

(This issue has bigger repercussions than simply the project I am working on. OS X El Capitan includes a feature that makes /usr/include permanently immutable, even by the root user. If this gem is not patched to accommodate this, it will become impossible to install on Macs that run OS X El Capitan.)

[flavorjones]

Hi, @wjk. Thanks for bringing up this issue.

I believe what you're referring to was introduced in commit 8ed4486 to address #1111. This change made it into Nokogiri 1.6.4 and later. The message you receive should be letting you know that you need to run xcode-select --install, have you done that? If not, why not? If so, where is your iconv.h located? Can you provide some more information about your system? What version of XCode are you running, what version of OSX are you running, are you running homebrew (and what version), etc.?

It sounds like you have a better idea to fix #1111 than what we came up with -- can you be more specific about how you think we should be trying to detect iconv?

I'd like to make the installation experience better on OSX, but unfortunately at this point the ecosystem has become quite fractured and it's hard to handle every edge case. We're actively looking for help from the community to make Nokogiri better, so any information or thoughts you have on how to do this would be valuable to us. We're also trying to make this page the canonical reference for how-to-install:

http://www.nokogiri.org/tutorials/installing_nokogiri.html

so once we figure out what's going on here, we'll make sure to update that page as well.

---

Now that I've hopefully demonstrated that nokogiri-core wants to help, I'd like to very kindly ask you to check the value judgements you're making ("bad design", "shortsighted") and the language of helpless entitlement you're using ("even if ... you decide to fix it, it won't help me much"). Nokogiri is an open-source project that's worked on by people in their free time, and who want to make the world better. If you describe the problem you're having and provide information that will help the core contributors diagnose and fix the issue, I promise we'll respond as quickly and helpfully as we can. Thanks for listening.

[flavorjones]

@wjk Also, if you could please provide the output from a failed gem install, as well as the mkmf.log file, that would help us further narrow down the problem and solution. Frequently, people will pile onto "failed to install" issues that are unrelated because there's not enough information in the issue to disambiguate.

Thanks again.

[wjk]

First, I would like to apologize for my heated tone above. I was very frustrated when I wrote that because my build just broke for the umpteenth time. I did not mean to take my anger out on you, or on your project. I’m sorry.

---

I do not want to run xcode-select --install on my system, because I frequently update Xcode.app and if I ran that command, the duplicate Xcode command-line tools would quickly become out of sync. I would recommend simply using the mkmf find_header function, which uses the compiler’s standard include paths to determine if iconv.h is present, rather than checking in one specific place only.

I am currently running Xcode 7 beta 5. On my system, iconv.h is located at /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/iconv.h. (To find it on your system, I would recommend running cd $(xcrun --show-sdk-path)/usr/include, which evaluates to the path to the /usr/include directory for the default Xcode SDK.) This path is one of the default locations searched by the copy of clang included with Xcode.

[flavorjones]

@wjk Thanks for following up. I think this is the first time we've heard pushback on installing the command-line tools, and I'm only a Linux user, but what you're saying seems to make sense.

@knu and I had a brief chat at the commit in question

8ed4486

and it sounds like perhaps this hack was in place only for older XCode versions.

@zenspider - can you think of any harm being done by either removing the hardcoded path, or by replacing it with an appropriate call to an mkmf helper?

[kylef]

Just want to point out #1218 is related, merging that would actually fix this issue.

[zenspider]

@flavorjones if @knu is OK with changing it, I'm OK with changing it.

[fxposter]

Any updates, nokogiri currently fails to compile on Mac OS X 10.11? /cc @knu

[knu]

I seem to have /usr/include/iconv.h on all of my Macs, so I can never do meaningful tests.
That said, the check was only to help diagnose problematic environments, so I think it's kind of safe to change it.

[flavorjones]

I think I'd like to get this into 1.6.7, since it seems to be turning into an "installation improvement" release. ;)

[netheril96]

I have the same issue and the same feeling as the author. I already installed Xcode.app which includes its own set of command line tools, and I don't want to install a duplicate set, which may go out of sync at any time. nokogiri is currently the only one failing to be built on my Mac now.

[flavorjones]

@RSY96 Thanks for your additional post.

I mentioned above that a fix is going into 1.6.7, no need to further convince anyone of anything.

[netheril96]

@flavorjones Sorry that I didn't read all the posts carefully. I just glanced at this long post and thought this was just another case where the issue was never going to be resolved.

[mojca]

I'm using OS X 10.7 (and I'm a newbie with rails). Just for completeness: I have full Xcode installed as well as MacPorts under /opt/local, but no HomeBrew and nothing under /usr/local[/lib]. I would like to avoid linking against MacPorts libraries as they get removed or renamed when they get updated.

After problems with libxml2 and reading some posts about recent changes I tried to install the latest version, but it didn't really help:

gem install nokogiri -v 1.6.7.rc3

I'm still experiencing problems with libxml2:

Running 'configure' for libxml2 2.9.2... OK
Running 'compile' for libxml2 2.9.2... OK
Running 'install' for libxml2 2.9.2... OK
Activating libxml2 2.9.2 (from /Users/me/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.rc3/ports/x86_64-apple-darwin11.4.2/libxml2/2.9.2)...
[...]
Extracting libxslt-1.1.28.tar.gz into tmp/x86_64-apple-darwin11.4.2/ports/libxslt/1.1.28... OK
Running git apply with /Users/me/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.rc3/patches/libxslt/0001-Adding-doc-update-related-to-1.1.28.patch... OK
[...]
Running 'configure' for libxslt 1.1.28... OK
Running 'compile' for libxslt 1.1.28... OK
Running 'install' for libxslt 1.1.28... OK
Activating libxslt 1.1.28 (from /Users/me/.rbenv/versions/2.2.3/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.7.rc3/ports/x86_64-apple-darwin11.4.2/libxslt/1.1.28)...
checking for main() in -llzma... no
checking for xmlParseDoc() in libxml/parser.h... no
checking for xmlParseDoc() in -lxml2... no
checking for xmlParseDoc() in -llibxml2... no
-----
libxml2 is missing.  Please locate mkmf.log to investigate how it is failing.
-----

One of the most weird part is this one though:

have_iconv?: checking for iconv... -------------------- yes

"clang -o conftest -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0/x86_64-darwin11.0 -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0/ruby/backward -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0 -I. -I/Users/me/.rbenv/versions/2.2.3/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe  conftest.c  -L. -L/Users/me/.rbenv/versions/2.2.3/lib -L. -L/Users/me/.rbenv/versions/2.2.3/lib  -fstack-protector -L/usr/local/lib     -lruby-static -framework CoreFoundation  -lpthread -ldl -lobjc  "
Undefined symbols for architecture x86_64:
  "_iconv", referenced from:
      _main in conftest-ik3fT0.o
  "_iconv_open", referenced from:
      _main in conftest-ik3fT0.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <stdlib.h>
 4: #include <iconv.h>
 5: 
 6: int main(void)
 7: {
 8:     iconv_t cd = iconv_open("", "");
 9:     iconv(cd, NULL, NULL, NULL, NULL);
10:     return EXIT_SUCCESS;
11: }
/* end */

"clang -o conftest -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0/x86_64-darwin11.0 -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0/ruby/backward -I/Users/me/.rbenv/versions/2.2.3/include/ruby-2.2.0 -I. -I/Users/me/.rbenv/versions/2.2.3/include  -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT    -O3 -Wno-error=shorten-64-to-32  -pipe  conftest.c  -L. -L/Users/me/.rbenv/versions/2.2.3/lib -L. -L/Users/me/.rbenv/versions/2.2.3/lib  -fstack-protector -L/usr/local/lib     -lruby-static -framework CoreFoundation -liconv -lpthread -ldl -lobjc  "
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <stdlib.h>
 4: #include <iconv.h>
 5: 
 6: int main(void)
 7: {
 8:     iconv_t cd = iconv_open("", "");
 9:     iconv(cd, NULL, NULL, NULL, NULL);
10:     return EXIT_SUCCESS;
11: }
/* end */

I can link to more logs if needed, but I'm not sure about the most suitable ticket. The libxml2 tests finally fail because of the wrong number of arguments in conftest:

conftest.c:15:27: error: too few arguments to function call, single argument 'cur' was not specified
int t(void) { xmlParseDoc(); return 0; }
              ~~~~~~~~~~~ ^

[bobashforth]

I'm running El Capitan on a Mac, and have my iconv.h header located in

Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/iconv.h.

I also have it located at
/usr/local/Cellar/libiconv/1.14/
since I installed libiconv using brew.

The above instructions do not work for me; I get the message:

"The file "/usr/include/iconv.h" is missing in your build environment,
which means you haven't installed Xcode Command Line Tools properly."

I've been through this on earlier OS versions, despite having installed XCode straight from the app store. (So I can't do xcode-select --installed)

Any suggestions?

Thanks-

Bob

[xymbol]

@bobashforth Please note that libiconv has been moved to the dupes repo.

[bobashforth]

Thanks, xymbol; that is the repo I tapped to install libiconv from brew. My issue is that I do have libiconv, and I do have iconv.h, and I do have Xcode installed (single copy), and installing Nokogiri fails, with any of the above suggestions duly followed.

It took me days, if not weeks, to get nokogiri installed on my work laptop originally; it's been swapped out for a new one, unfortunately not preserving my toolset, and I'm in the same boat now as I was two years ago. :-(

Again, any suggestions?

Thanks-

Bob

[xymbol]

@bobashforth I hear you. Have Xcode 7.0.1 installed on 10.11 and the new solution didn't work either. Had to install Command Line Tools on top of Xcode, which I didn't want to. It's working for me now.

[Sam-Gram]

I got nokogiri install on El Capitian by following this: http://stackoverflow.com/questions/5528839/why-does-installing-nokogiri-on-mac-os-fail-with-libiconv-is-missing

[flavorjones]

Please note that #1218 has been merged, which we believe will address this issue. Would love for someone to try building from source ("bundle exec rake compile") and let us know.

[rjsathome]

Sorry, but I also have to tell that I quite often run into troubles when I need nokogiri.

This time I try to setup a rails app on a fresh MacBook with El Captain on it. I installed the latest Xcode

Rainers-MBP (2)$ xcodebuild -version
Xcode 7.1.1
Build version 7B1005

This time I run into this problem:

tail /Users/rjs/.rvm/gems/ruby-2.2.3@kdb/extensions/x86_64-darwin-14/2.2.0-static/nokogiri-1.6.6.4/mkmf.log

"gcc -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0/x86_64-darwin14 -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0/ruby/backward -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT   -O3 -I/Users/haven/.sm/pkg/active/include -fPIC -mmacosx-version-min=10.7 -pipe  -Wno-error=unused-command-line-argument-hard-error-in-future  -arch x86_64  -Werror -c conftest.c"
error: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused-command-line-argument'? [-Werror,-Wunknown-warning-option]
checked program was:
/* begin */
1: #include "ruby.h"
2:
3: int main() {return 0;}
/* end */

[rjsathome]

Here my tricks how it finally worked:

• the iconv problem is still there, following all the tips from nokogiri did not help, but this tips did: http://stackoverflow.com/questions/5528839/why-does-installing-nokogiri-on-mac-os-fail-with-libiconv-is-missing
• then working around the "-Werror=unused-command-line-argument-hard-error-in-future" problem by editing the extconf.rb locally and running the build manually it helped, here the final steps:

  171  /Users/rjs/.rvm/rubies/ruby-2.2.3/bin/ruby extconf.rb --with-iconv-dir=/usr/local/Cellar/libiconv/1.14
  172  vi extconf.rb

  177  /Users/rjs/.rvm/rubies/ruby-2.2.3/bin/ruby extconf.rb --with-iconv-dir=/usr/local/Cellar/libiconv/1.14
  178  make
  179  make install

[zenspider]

On Nov 24, 2015, at 13:30, rjsathome [email protected] wrote:

Sorry, but I also have to tell that I quite often run into troubles when I need nokogiri.

This time I try to setup a rails app on a fresh MacBook with El Captain on it. I installed the latest Xcode

Rainers-MBP (2)$ xcodebuild -version
Xcode 7.1.1
Build version 7B1005

This time I run into this problem:

tail /Users/rjs/.rvm/gems/ruby-2.2.3@kdb/extensions/x86_64-darwin-14/2.2.0-static/nokogiri-1.6.6.4/mkmf.log

"gcc -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0/x86_64-darwin14 -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0/ruby/backward -I/Users/rjs/.rvm/rubies/ruby-2.2.3/include/ruby-2.2.0 -I. -D_XOPEN_SOURCE -D_DARWIN_C_SOURCE -D_DARWIN_UNLIMITED_SELECT -D_REENTRANT -O3 -I/Users/haven/.sm/pkg/active/include -fPIC -mmacosx-version-min=10.7 -pipe -Wno-error=unused-command-line-argument-hard-error-in-future -arch x86_64 -Werror -c conftest.c"
error: unknown warning option '-Werror=unused-command-line-argument-hard-error-in-future'; did you mean '-Werror=unused-command-line-argument'? [-Werror,-Wunknown-warning-option]
checked program was:
/* begin /
1: #include "ruby.h"
2:
3: int main() {return 0;}
/ end */

xcode doesn't ship with gcc, but you're using gcc to build. I suspect something is out-of-date with one of your rvm builds.

[rjsathome]

Oh, I'm not using gcc.

I just tried to do a gem install nokogiri something behind the scenes decides to use gcc.

Probably one of the tries to follow nokogiri's tips how to fix things (probably installing the command line tools with xcode-select --install placed some gcc in the system that is found by nokogiri. But behind the scenes it is still llvm.

Rainers-MBP (1)$ type gcc
gcc is hashed (/usr/bin/gcc)
Rainers-MBP (1)$ gcc -v
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 7.0.0 (clang-700.1.76)
Target: x86_64-apple-darwin15.0.0
Thread model: posix

[flavorjones]

@rjsathome If you're having a problem, please open a new issue, and please don't post to a closed ticket. The problem you're describing is not the same as the OP's problem.

Locking this ticket down.