Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release statically linked protoc for Linux #5904

Closed
wants to merge 5 commits into from

Conversation

anuraaga
Copy link
Contributor

@anuraaga anuraaga commented Mar 16, 2019

There was some interest in a statically linked protoc in #5053. Statically linked binaries can prevent loader issues when running protoc such as in the linked issues while making the build process simpler, as not as much care has to be taken in using dependencies outside of system dependencies. This means that modern OS distributions can be used to build protoc instead of intentionally using an old one to link to old versions of libc.

Pros of static link

  • All Linux users will be able to run the binary, no matter what distribution
  • No restrictions on build environment - an OS with modern GCC and libc can be used without issue since the resulting binaries are loadable anywhere

Cons of static link

  • Larger binary. Comparing 3.7.0, a statically linked one is 5.7MB vs 4.7MB for the released dynamically linked one

I didn't see any strong objection to statically linking in #5053 and it seemed to get closed since the immediate issue at hand was solved. But as we've seen with libatomic, these issues will continue to crop up as long as the binaries are dynamically linked. Hopefully this PR makes sense - if there is a reason to avoid statically linking though, feel free to close it. Merging and using it will fix the libatomic issue though :) The best part of it, IMO, is that the same process is used to build all the targets - currently, aarch64 and ppc64 are built on the Kokoro debian host and others built in a CentOS docker container. This discrepancy doesn't seem intuitive or ideal. I suspect the libatomic release issues happened in the first place since developers use debian, which includes libatomic automatically when installing a build chain, and not centos.

Changes

  • Switch build image from old centos to modern debian, which includes good support for cross-compiling all the target architectures
  • Pass --static to LDFLAGS on linux, ensuring resulting binary is not a dynamic executable
  • Update dependency checker to verify the linux binaries are statically linked
  • Kokoro script uses the same docker image to build all Linux targets

Testing

I ran all builds using the changed Dockerfile using something like

$ docker build -t protobuf/build protoc-artifacts
$ docker run -it --rm protobuf/build
$ apt-get install -y git
$ git clone https://github.com/anuraaga/protobuf
$ git fetch origin static-link
$ git checkout static-link
$ ./autogen.sh
$ cd protoc-artifacts
$ ./build-protoc.sh linux x86_64 protoc
$ ./build-protoc.sh linux x86_32 protoc
$ ./build-protoc.sh linux aarch_64 protoc
$ ./build-protoc.sh linux ppcle_64 protoc
$ file target/linux/aarch_64/protoc.exe
$ file target/linux/ppcle_64/protoc.exe

All the built binaries ran fine. The aarch / ppcle ones I think have the qemu runtime layer statically linked in, but the binaries seem to not be much larger as a result so hopefully this is fine.

Fixes #5875

@anuraaga
Copy link
Contributor Author

@acozzette Would be great if you could consider this as a solution for #5875

@acozzette
Copy link
Member

@anuraaga Thanks so much for working on this but I'm a bit hesitant to accept this change, mostly because I have heard that statically linking libc can be unreliable. Some people seem to say glibc in particular is not safe to link against statically. e.g. here. Maybe this is something we should consider again in the future but now that I found a small fix for the libatomic issue in #5922 I think we should probably stick with that for now. Producing protoc binaries that work portably has definitely been a source of problems, so sorry about the trouble and thanks again for looking at this.

@TeBoring TeBoring requested review from acozzette and removed request for TeBoring March 26, 2019 21:30
@TeBoring TeBoring assigned acozzette and unassigned TeBoring Mar 26, 2019
@acozzette acozzette closed this Apr 8, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Released protoc 3.7.0 require libatomic1 (not statically linked?)
4 participants