Binaries for sunos do not work on Solaris 11.2 - 'GLIBCXX_3.4.20' not found

[GeorgeBailey]

We're increasing our use of Solaris 11.2 X86. The only snag right now is staying compatible with the latest Node.JS releases.

wget http://nodejs.org/dist/v4.2.0/node-v4.2.0-sunos-x64.tar.gz
tar xoZf node-v4.2.0-sunos-x64.tar.gz
node-v4.2.0-sunos-x64/bin/node

I get this for both x86 and x64 binaries.

ld.so.1: node: fatal: libstdc++.so.6: version 'GLIBCXX_3.4.20' not found (required by file node-v4.2.0-sunos-x64/bin/node)
ld.so.1: node: fatal: libstdc++.so.6: open failed: No such file or directory
Killed

Edit: We can now readily build from src on Solaris 11.2 with minor tweaks after downloading v4.2.1.

This issue should now only reference the desire of a pre-compiled that is compatible with Solaris 11.2.

[mscdex]

/cc @nodejs/platform-solaris

[alexijs]

It may not be the best solution, however you can tell the runtime linker to ignore the shared library versions.

SunOS prod 5.11 11.2 i86pc i386 i86pc

IT DOES SIMPLY START IT AND NOT ALLOWING NETWORK/SOCKET CONNECTIONS

[GeorgeBailey]

I can confirm that adding LD_NOVERSION=1 at runtime as @alexijs suggested allows startup, but results in EPROTOTYPE error on TCP or UNIX socket functions. (I did successfully use child processes, crypto.randomBytes.)

Referring to my previous issue, I was getting EPROTOTYPE in v0.12 pre-built binaries as well. This was resolved by cgalibern in a fork. (though I was still getting ECONNREFUSED on outbound TCP)

I wonder whether the same change would resolve EPROTOTYPE in v4.2.0 or if it is a different issue entirely.

[GeorgeBailey]

On mine, I used pkg install gcc which through the dependency tree includes gnu-binutils and I believe it also includes gnu-make. I don't know which compiler is the best to use, but what I really want is to be compatible with both Solaris 10 and Solaris 11. :-)

I don't know what Solaris 10 compatibility would entail, so let's just get Solaris 11 compatibility for now.

[jbergstroem]

@alexijs: CC=/usr/bin/gcc ./configure should work fine if you want to switch compiler

[misterdjules]

There are two different categories of problems mentioned in this issue:

1. Running the binaries downloaded from nodejs.org on Oracle Solaris is not possible due to the proper version of the C++ runtime library not found (the 'GLIBCXX_3.4.20' not found message).
2. Building nodejs/node's master and v4.x from source doesn't work on Oracle Solaris.

2. is tracked with node compile failure on Solaris 11.2 and gcc 4.8 (undefined std::isnan) #3179, so please comment in that issue for any problem related to building node on Solaris.

It would be great if we could also solve 1). Let's see what happens on SmartOS.

SmartOS packages for node v4.x require a C++ runtime at version GLIBCXX_3.4.15:

[root@dev ~]# pvs `which node`
        libkstat.so.1 (SUNW_0.7);
        libumem.so.1 (SUNW_1.1);
        libsocket.so.1 (SUNW_1.7);
        libnsl.so.1 (SUNW_1.7, SUNWprivate_1.1);
        librt.so.1 (SUNW_1.4);
        libsendfile.so.1 (SUNW_1.1);
        libstdc++.so.6 (GLIBCXX_3.4.15, CXXABI_1.3);
        libm.so.2 (SUNW_1.2);
        libgcc_s.so.1 (GCC_3.0);
        libpthread.so.1 (SUNW_1.2);
        libc.so.1 (SUNW_1.23, SUNWprivate_1.1);
[root@dev ~]#

This is definitely older than GLIBCXX_3.4.20, so it has higher chances to run on more systems. The node binary links against /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libstdc++.so.6, and not against /usr/lib/libstdc++.so.6:

[root@dev ~]# ldd `which node`
        libz.so.1 =>     /opt/local/lib/libz.so.1
        libkstat.so.1 =>         /lib/64/libkstat.so.1
        libumem.so.1 =>  /lib/64/libumem.so.1
        libsocket.so.1 =>        /lib/64/libsocket.so.1
        libnsl.so.1 =>   /lib/64/libnsl.so.1
        librt.so.1 =>    /lib/64/librt.so.1
        libsendfile.so.1 =>      /lib/64/libsendfile.so.1
        libstdc++.so.6 =>        /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libstdc++.so.6
        libm.so.2 =>     /lib/64/libm.so.2
        libgcc_s.so.1 =>         /opt/local/gcc48/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1
        libpthread.so.1 =>       /lib/64/libpthread.so.1
        libc.so.1 =>     /lib/64/libc.so.1
        libgcc_s.so.1 =>         /opt/local/gcc47/x86_64-sun-solaris2.11/lib/amd64/libgcc_s.so.1
        libmp.so.2 =>    /lib/64/libmp.so.2
        libmd.so.1 =>    /lib/64/libmd.so.1
[root@dev ~]#

And that C++ runtime is compatible with version GLIBCXX_3.4.15:

[root@dev ~]# nm /opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19 | grep GLIBCXX | grep 3.4.15
00000000 A GLIBCXX_3.4.15
[root@dev ~]#

I don't know yet what makes the pkgsrc packages on SmartOS link with that specific version of libstdc++ and not the default one, I'd need to investigate further.

And finally, SmartOS pkgsrc packages specifically use GCC 4.8 to build the node binary, whereas it seems that the Node.js release jobs use GCC 4.9.1. This helps make the resulting node binary require an older C++ runtime version.

In other words, it is very likely that the Node.js release jobs could make the resulting node binaries link against an older C++ runtime, which could make these binaries run on Oracle Solaris 11.2 by default.

Just to give more context, at some point a similar issue was encountered for Linux binaries distributed by nodejs.org. To work around this problem, Linux node binaries released on nodejs.org use the RedHat developer toolset to link against a specific C++ runtime that includes a static part with newer C++ symbols so that the dynamic part of the C++ runtime can be found on older system, and the node binary can use newer C++ features by including them directly in the binary.

Unfortunately I don't have a Solaris setup at hand, and that makes testing these theories and possible fixes difficult, if not impossible. But that's a starting point for anyone who would be willing to tackle that problem. I might get some time to work on that in the future, in which case I'll keep you posted.

[alexijs]

@jbergstroem: this should work. However, I believe, it gotta be lower case cc.
I relinked them back to solarisstudio compilers (I build apache and php using them and solaris optimization flags without a problem)

[misterdjules]

I don't know yet what makes the pkgsrc packages on SmartOS link with that specific version of libstdc++ and not the default one, I'd need to investigate further.

Actually, just using the GCC 4.8 found in SmartOS pkgsrc packages make the binary link against /opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19.

[misterdjules]

@GeorgeBailey Also, the --without-mdb configuration option is not present in node v4.x and later. mdb_v8 is not bundled with node v4.x and later, because bundling the mdb_v8 debugger module in the node binary is actually not useful (it was at some point). Instead, the recommended way to use it is to download the latest release.

[GeorgeBailey]

Have you compiled it successfully via gnu utils?

@alexijs, I was able to compile v0.12, but not v4.2.0.

There are two different categories of problems mentioned in this issue:

@misterdjules, I will start using #3179 for issues building. (your # 1) It makes sense to keep this separate.

So this issue should only be for Solaris 11.2 compatibility with pre-built binaries.

An idea - it would be relevant to know what build environment is used. Is there a wiki on this, so that another person could recreate the build environment on their own?

[misterdjules]

An idea - it would be relevant to know what build environment is used. Is there a wiki on this, so that another person could recreate the build environment on their own?

Definitely. The nodejs/build repository already has some (possibly outdated) documentation. I'm wondering if the best documentation in this case is the source, for instance: https://github.com/nodejs/build/blob/master/setup/smartos/ansible-vars.yaml.

[misterdjules]

@GeorgeBailey @jbergstroem Speaking of https://github.com/nodejs/build/blob/master/setup/smartos/ansible-vars.yaml, I would start investigating replacing gcc49 and gcc49-libs by gcc48 and gcc48-libs and see what impact it has on the C++ runtime that the node binary links to. Chances are that such a binary would work on Solaris 11.2.

[jbergstroem]

@misterdjules: only gcc47 and gcc49 are available in 2014Q3 (which our bots are running). Time to update?

[misterdjules]

@jbergstroem I would suggest testing this with 2014Q4, which is a LTS release that has gcc 4.8. But I'd like to confirm with @jperkin before moving forward to avoid wasting time.

@jperkin IIRC, 2014Q4 doesn't have OpenSSL 1.0.2, but in this case it should not be a problem since node would be built with the bundled OpenSSL in deps/.

[alexijs]

@GeorgeBailey: let me advice you on compatibility with sol v10 and v11: you should compile it on v10 and you may be transfer the binary to the same arch on v11. You wont be able to do it vice versa.
However, I would suggest compile it separately for different platforms, make a package, and add it to your internal repos for each os version.

[misterdjules]

@alexijs Thank you very much for sharing your progress, it is very much appreciated. Do you mind continuing that discussion in #3179 though? This issue is about not being able to run binaries available on nodejs.org, and discussing build issues just makes things very confusing.

[misterdjules]

Regarding running v4.x binaries available on nodejs.org on Solaris 11.2, here's what I think is an accurate summary of the current situation:

1. Binaries available on nodejs.org are built using gcc/g++ 4.9 because:
   • The build fails with gcc/g++ 4.7.x.
   • g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.
2. It would be possible to upgrade build machines to a pkgsrc version that provides gcc/g++ 4.8 and build sunos binaries with this toolchain. Unfortunately, that means node would be dynamically linked to a C++ runtime with a path that is not the default path (/opt/local/gcc48/x86_64-sun-solaris2.11/lib/libstdc++.so.6.0.19). Thus, any user who would want to run node on sunos would need to install that dynamic library at the same path by installing the optional gcc-4.8 package on SmartOS. I'm not sure what would be the impact on Solaris 11.2. So that would break even more users than building node binaries with gcc/g++ 4.9.
3. The reason why node v4.x doesn't build with g++ is because of a g++ bug whose fix hasn't been backported to g++ 4.7.x. However, it seems that node v4.x can actually be built and passes all tests with g++ 4.7 with a simple patch.
4. We could also explore other options, such as providing the C++ runtime library in the binary tarball, or having a fully static node binary on sunos, but both these potential solutions seem overkill compared to 3).
5. There's also the option of letting package managers handle these differences, but again, 3) seems non-intrusive enough to be worth trying.

@nodejs/build @nodejs/collaborators As mentioned above, 3) seems like it would not result in any regression, and it could allow more Solaris users to use v4.x binaries from nodejs.org. Thoughts on trying that approach?

[evanlucas]

I think that opening a PR with the patch will be the best way to get actual feedback on whether that patch is acceptable (which doesn't appear to be a major change) would be the best step. Maybe /cc @indutny ?

[misterdjules]

@evanlucas Good point, will do that soon, thank you!

[misterdjules]

@evanlucas Done with #3391.

[alexijs]

@misterdjules wrote: 'g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.'
Why are they not available? Its latest supported by Oracle version.
May be it makes sense to build Node for Solaris and Smart OS separately for each OS?
I think calling a smartOS package a solaris package is too misleading...

Also, many companies release binaries for each OS version separately. I thinks its a very good practice especially from QA standpoint.

In ideal world there would be node-sol-10.pkg, node-sol-11.pkg, and node-smartos.pkg
And in the perfect world there would be a node repos for online installation.

[alexijs]

@misterdjules: I moved compilation posts to the topic you suggested.

[misterdjules]

@alexijs Excellent thank you!

@misterdjules wrote: 'g++ 4.8 is not available on the build machines currently used to build release binaries for sunos.'
Why are they not available?

Because the build machines use a version of pkgsrc that is tool old to include g++ 4.8.

May be it makes sense to build Node for Solaris and Smart OS separately for each OS?
I think calling a smartOS package a solaris package is too misleading...

Indeed, that seems like a good suggestion, probably worth exploring.

[jbergstroem]

I'm happy to update our build toolset to 4.8, I just want to exhaust the discussion in #3391 first.

[rvagg]

Could we get some testers on the sunos binaries @ https://nodejs.org/download/nightly/v5.0.1-nightly2015111384bb74547d/ please? I have no idea if we're any closer to rectifying this but it's a new build setup.

[jelmd]

/tmp/node-v5.0.1-nightly2015111384bb74547d-sunos-x64 > bin/node
ld.so.1: node: fatal: libc.so.1: version 'ILLUMOS_0.5' not found (required by file bin/node)
ld.so.1: node: fatal: libc.so.1: open failed: No such file or directory
Killed
(S11.3)

[GeorgeBailey]

Same here on Solaris 11.2, both x86 and x64 binaries.

[rvagg]

Sorry, after actually reading some of the content above, particularly the details posted by @misterdjules, I realise this is about solaris vs smartos/illumos and not a smartos version issue which is what I had in my head. The binaries from that nightly would be interesting to test on older versions of smartos but there's nothing in there for vanilla solaris.

[misterdjules]

Here's a work in progress that describes exactly what's going on: https://gist.github.com/misterdjules/eae9ec70dc1d91fb8dd1. Comments and questions welcome. We'll use that document as the basis for improving our SmartOS builds, and also to determine what we can do for Solaris binaries.

[ApolonTorq]

I'm looking at using Node.js on OpenIndiana and/or Solaris 11.x regularly in the future.

The 5.0.1 nightly2015111384bb74547d build of Nodejs resolves the problem for the Hipster version (2015.10 illumos-7fc68dd) but not on the Legacy release (oi_151a8).

Later releases/builds don't appear to include the fix present in the 5.0.1 nightly build referenced. This includes the 6.0.0 series.

How can I help in ensuring Node.js runs "out of the box" in the future on Open Indiana?

[jbergstroem]

The plan (at least my plan, hehe) is to get access to a wider range of sunos derivatives and figure out the lowest common denominator in terms of toolchain that works on a wide range of deployments. I think its safe to say that smartos and sunos could benefit from two sets of releases though. Guessing Openindiana and smartos would play just fine together.

[misterdjules]

figure out the lowest common denominator in terms of toolchain that works on a wide range of deployments

I'm concerned that this approach would work mostly by luck (as it's been the case for current sunos builds for various solaris-derivatives platforms as demonstrated by https://gist.github.com/misterdjules/eae9ec70dc1d91fb8dd1) and would fail unexpectedly sometimes.

In general it seems to me that leaving the work of providing node binaries to packagers would make more sense on most platforms: they know the platform best to choose the most appropriate toolchain and build settings.

[misterdjules]

@ApolonTorq What errors do you get precisely? Do you mind creating a separate issue with a detailed description of the problem (error messages for each version, etc.)?

[ApolonTorq]

@misterdjules The errors are different unresolved shared library references for the different platforms. I'll create separate issues as requested. Note that if we decide to go with the latest Open Indiana hipster version then the shared library issues don't occur for both the current V5.5.0 and the latest nightly builds.

[jbergstroem]

Seeing how this hasn't made much progress as of late I just wanted to give an update. The build group still hasn't gotten access to solaris vm's. If any company and/or provider would step up it would likely make things go considerably faster. If you're interested, ping us over at https://github.com/nodejs/build.

[No9]

So @ptribble has just announced some great work on the solaris build issue
https://twitter.com/ptribble/status/728954509146378240
The actual builds are available here http://petertribble.co.uk/Solaris/node.html
Would be great if we could utilize some of this work more formally

[ptribble]

Just a commentary.

I've been shipping Solaris 10 packages for almost 5 years; my build machine is, fortunately, not yet at death's door.

On my Solaris 10 builds I ship the matching gcc libs from the compiler I use. (This is essential on Solaris 10 as the shipped gcc is 3.4.3 and completely ancient. But in the general case you simply can't trust the environment where the package is to be deployed to have matching versions.)

Labelling the binary as a SunOS build is incorrect, and leads to incorrect expectations by users.

I wouldn't ever have a reasonable expectation that an illumos built node would run on Solaris 11 (or vice-versa). They have 5 years of divergence. Something from Solaris 10 should run on both, but in this case lacks a few features, and it would be better to build native. Within both the Solaris 11 and illumos branches, you still have to build on something that's a few revisions back to allow binary compatibility to do its job. On the illumos side, a vanilla OmniOS r151014 might be a good base.)

[jbergstroem]

@ptribble: we've inherited the smartos/sunos build from the old nodejs org. I agree that naming it sunos is incorrect but seeing how we don't have any solaris hardware/vm to build on/support we've decided to not make any changes. If we get resources we will likely revisit and rename.

[jelmd]

Not sure, how you build it, but if you do it on an Illumos based Distro, why not name it this way (i.e. -${distro}-). This way one knows, where the binary package definitely should work. For all others it is a hint, may be or may be not ...

[MaheshVelankar]

@ptribble:
Peter,

I followed your instructions on this page.
http://www.petertribble.co.uk/Solaris/node.html

I tried V7 current version

Successfully bunzip-ed the package and installed it with pkgadd command as you have instructed.

Added /opt/Node/bin to path and tested the following

which node
/opt/Node/bin/node

and

which npm
/opt/Node/bin/npm

That confirms that PATH is good.

however the following is failing
npm -v
/usr/bin/env: Invalid argument

Please let me know how to proceed with this.

My solaris is

uname -a
SunOS 5.10 Generic_150400-35 sun4v sparc sun4v

Please let me know if I should provide more details.

Thanks
-Mahesh

[ptribble]

SunOS 5.10 Generic_150400-35 sun4v sparc sun4v

That's a SPARC system, which isn't supported by Node.JS at all, so that's an entirely different issue.

[michael-ts]

@MaheshVelankar Wait, where did you find an npm binary? v6 and beyond (at least) don't claim to have an npm binary, and there certainly isn't one in the bin directory.

@ptribble Thanks for the binaries, the system I'm using has a gcc way too old (4.4.4) to compile any decently recent node stuff! The node binary works great, although I have found that things like os.networkInterfaces() don't work properly. But how do you get npm to work? The only npm binary I see is in lib/node_modules/npm/bin but when I try to run it I get the error "Error: Cannot find module '/u/home/michael/node/lib/node_modules/npm/bin/node_modules/npm/bin/npm-cli.js'". But npm-cli.js is in the same directory as the npm binary! I've only got '/u/home/michael/node/bin' in my PATH, which is where the node binary is. (I used pkgtrans to extract the node package under my home directory because I don't have write access to /opt.)

[michael-ts]

@ptribble It seems that "node lib/node_modules/npm/bin/npm-cli.js" works just fine in place of "npm", although that's a weird way to invoke npm.

[ptribble]

@michael-ts You're almost there. The way svr4 packages work is that they don't store links, those are created during package installation but will be missing if you just do pkgtrans.

You can see what symbolic links should be present by looking for lines in the package's pkgmap file that have 's' in the second field.

And the missing symlink that's relevant here is

/opt/Node/bin/npm=../lib/node_modules/npm/bin/npm-cli.js

so if you symlink npm to the lib/node_modules/npm/bin/npm-cli.js file everything should work.

(And yes, the os.networkInterfaces() stuff doesn't work, because Solaris 10 plain doesn't have the calls to support it and that's what my packages are built on.)

[jasnell]

@nodejs/build ... does this need to remain open?

[ptribble]

I think the remaining issue here is that the downloadable binaries are labelled as "sunos" whereas they're actually specific to smartos and won't run on most of the other sunos variants. If they were labelled as smartos then people on other platforms would have their expectations set correctly.

The need for binary downloads is far less relevant now. Node is shipped with OpenIndiana and some other distros, builds pretty much out of the box on most current illumos distros and Solaris 11, and anyone still using Solaris 10 can use my builds.

[misterdjules]

@ptribble

I think the remaining issue here is that the downloadable binaries are labelled as "sunos" whereas they're actually specific to smartos and won't run on most of the other sunos variants. If they were labelled as smartos then people on other platforms would have their expectations set correctly.

See nodejs/build#500.

[jelmd]

+1

FWIW: We also rebuild nodeJS on demand for the latest pub. avail. amd64 Solaris 11 version (dyn linked against sys libs). The related native package repo is public as well: http://pkg.cs.ovgu.de/lnf/en/catalog.shtm ...

[hunger1]

jelmd, looking at http://pkg.cs.ovgu.de/lnf/en/index.shtmll and after searching for node, I did this:

pkg install -g http://pkg.cs.ovgu.de/lnf pkg://lnf/runtime/[email protected],5.11-1.3.14.6:20161123T205411Z

and it worked, but it is installed under "/local" rather than "/". I apologize for my ignorance, but how do I tell it to install under "/" or perhaps "/opt/inf"?

It would be nice to have node 6.11.3. I successfully built that version manually using:

CC=gcc ./configure

I'm not sure what to contribute to the maintainers so that would not be necessary, I had to do that as otherwise it used "cc" for the assembler code and of course the Solaris C compiler does not understand GNU assembly language files.

[jelmd]

@hunger1
Yes, that's the biggest disadvantage wrt. IPS (solaris 11+ packaging): packages are not relocatable anymore, even if the binaries/scripts are. Because we use /local/usr etc. as the NFS share for our clients similar to what others use as /usr/local, we prefix software shared a priori by all clients with /local.

One way to get around it would be to loopback mount /opt/inf to /local/usr . Another option is to simply re-package the package with a different path. Finally and not very nice, you might find a SVr4 (and thus relocatable) package here - for details have a look at the README.txt!

Wrt. 6.11.3: I can give it a try. (Tried 6.11.2 some weeks ago, but it has some test issues (threading, encryption) ...).

Yes, use GNU compilers for node (like for "bootstrapping" of most other languages). Shouldn't make a big difference wrt. Studio Compiler on x86 (which uses g++ stuff as backend for all C++ stuff anyway IIRC). Also note, that with CC=gcc ./configure you probably build your own local zlib + openssl libs + libuv which in turn get linked statically to the node binary. Therefore node does not benefit from any system updates of related packages ... ( FWIW and completeness: all other patches I use )

[gireeshpunathil]

is this still outstanding?

[gireeshpunathil]

inactive for long, closing.