Error linking libzstd.so on ubuntu 19.04

[marianoguerra]

building this repo or a project with it as a dependency on ubuntu 19.04 I get this error:

   Compiling rdkafka v0.21.0 (rust-rdkafka)
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"
  = note: /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_decompress':
          rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:44: undefined reference to `ZSTD_getFrameContentSize'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:91: undefined reference to `ZSTD_getErrorCode'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:80: undefined reference to `ZSTD_decompress'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:82: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:100: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/target/debug/deps/librdkafka_sys-3d04a478eb2ea7f3.rlib(rdkafka_zstd.o): in function `rd_kafka_zstd_compress':
          rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:130: undefined reference to `ZSTD_compressBound'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:141: undefined reference to `ZSTD_createCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:155: undefined reference to `ZSTD_initCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:157: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:168: undefined reference to `ZSTD_compressStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:169: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:170: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:158: undefined reference to `ZSTD_getErrorName'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:198: undefined reference to `ZSTD_endStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:199: undefined reference to `ZSTD_isError'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:213: undefined reference to `ZSTD_freeCStream'
          /usr/bin/ld: rust-rdkafka/rdkafka-sys/librdkafka/src/rdkafka_zstd.c:200: undefined reference to `ZSTD_getErrorName'
          collect2: error: ld returned 1 exit status


error: aborting due to previous error

error: Could not compile `rdkafka`.

if I copy the command in note: and add "-lzstd" it seems to link

from

"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil"

to

"cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" [...] "-Wl,-Bdynamic" "-lz" "-lutil" "-lutil" "-ldl" "-lrt" "-lpthread" "-lgcc_s" "-lc" "-lm" "-lrt" "-lpthread" "-lutil" "-lutil" "-lzstd"

here is a Dockerfile to reproduce:

FROM ubuntu:19.04

ARG UNAME=mariano
ARG UID=1000
ARG GID=1000
RUN groupadd -g $GID -o $UNAME
RUN useradd -m -u $UID -g $GID -o -s /bin/bash $UNAME

RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update; \
  apt-get -y upgrade && \
  apt-get install -y --no-install-recommends wget build-essential clang libclang-dev libssl-dev zlib1g-dev libsasl2-dev libzstd-dev curl ca-certificates git

USER $UNAME
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y

then

# build it
sudo docker build --build-arg UID=$$(id -u) --build-arg GID=$$(id -g) -t ubuntu-rust:1904-stable ubuntu-1904-rust

sudo docker run -i ubuntu-rust:1904-stable sh << COMMANDS
    cd
    . .cargo/env
    git clone https://github.com/fede1024/rust-rdkafka.git
    cd rust-rdkafka
    cargo run --example simple_consumer
COMMANDS

it seems for some reason it doesn't find libzstd

$ ldconfig -p | grep libzstd

        libzstd.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libzstd.so.1

I tried creating symbolic links in /usr/local/lib just in case but it didn't work.

[marianoguerra]

adding a build.rs file with

fn main() {                                                                    
    println!("cargo:rustc-link-lib=zstd");
} 

and adding to Cargo.toml in the package section:

build = "build.rs"

makes it work, but I think there should be a better solution

[tristan]

I just ran into a similar issue when trying to use rust-rdkafka as a dependency on an archlinux system.

e.g.

[dependencies.rdkafka]
version = "~0.21"
features = ["ssl", "sasl"]

I couldn't fix this with a build.rs file because that only effects my own code and not the dependency.

to fix it i had to set RUSTFLAGS='-C link-args=-lzstd' before building, which adds -lzstd to the compilation of every dependency in my project.

[ghost]

Running into this on macos mojave as well. the build.rs solution fixed it.

[scrogson]

On macOS I was able to get around this by adding the following to a .cargo/config at the root of my crate:

[target.x86_64-apple-darwin]
rustflags = [
    "-C", "link-arg=-undefined",
    "-C", "link-arg=dynamic_lookup",
]

I assume one might try a similar technique in linux. Maybe replace the flags above to just:

"-C", "link-arg=-lzstd"

[madgene]

Thanks, @marianoguerra. You saved the day. I have to add "static" to make it work in my case, though.

fn main() {
    println!("cargo:rustc-link-lib=static=zstd");
}

[ghost]

Is it possible to somehow link it to zstd-sys?

[ghost]

For Archlinux you need to install zstd-static from the AUR to make it work with the build.rs hack. (the regular package only provides the dynamic library)

[ghost]

0efbcf3 Fixes this for me!

[benesch]

Urgh, it's unfortunate that we're now forcing everyone to take a dependency on libzstd. This is actually an upstream problem, as librdkafka's build system forces you to depend on libzstd: confluentinc/librdkafka#2305.

Once that lands I'd like to put the support for libzstd behind a feature flag, unless someone objects.

[benesch]

For those following along: confluentinc/librdkafka#2305 has landed, so as soon as that makes it into an upstream release, we can make zstd support conditional.

[benesch]

Since #163, zstd support is now properly conditional. Closing this out, as there's nothing left to do!