1.4.0: Building apps fails on latest alpine, when using bundled librdkafka #454
Comments
bschofield
changed the title
|
similar issue here from golang:1.14.2 I also can't install librdkafka 1.4.0 from deb using this manual https://docs.confluent.io/current/installation/installing_cp/deb-ubuntu.html#systemd-ubuntu-debian-install Result in: |
|
Thank you for reporting this. The bundled librdkafka alpine builds is built on Alpine 3.8. |
|
@bschofield |
|
@otavioprado Since you are on Debian you should not see the link error, can you provide the full output (including build command) that fails your Go application build? |
|
@edenhill Thanks for the fast response! I did not use Would it be useful to include a comment about the need to add |
|
Along these same lines, I can confirm that this worked with ...
RUN CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build -tags static_all,netgo -o /bin/app ./cmd/**/main.go
# final stage
FROM alpine:3.10
COPY --from=builder_base /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY --from=builder_base /bin/app /bin/appBut as soon as I switch to 1.4.0 of confluent-kafka-go, and librdkafka, I get the below error: Here's the base image with the updated versions... For the working version, just substitute the docker image to FROM golang:1.14.2-alpine3.11 AS go_kafka_base
RUN apk update && apk add --no-cache \
bash \
build-base \
coreutils \
gcc \
git \
make \
musl-dev \
openssl-dev \
openssl \
libsasl \
libgss-dev \
rpm \
lz4-dev \
zlib-dev \
ca-certificates \
wget && \
cd $(mktemp -d) && \
wget -nv -O cyrus-sasl-2.1.27.tar.gz https://github.com/cyrusimap/cyrus-sasl/releases/download/cyrus-sasl-2.1.27/cyrus-sasl-2.1.27.tar.gz && \
tar -xz --strip-components=1 -f cyrus-sasl-2.1.27.tar.gz && \
rm -f cyrus-sasl-2.1.27.tar.gz && \
./configure --prefix=/usr --disable-sample --disable-obsolete_cram_attr --disable-obsolete_digest_attr --enable-static --disable-shared \
--disable-checkapop --disable-cram --disable-digest --enable-scram --disable-otp --disable-gssapi --with-dblib=none --with-pic && \
make && \
make install
RUN cd $(mktemp -d) && \
wget -nv -O v1.4.0.tar.gz https://github.com/edenhill/librdkafka/archive/v1.4.0.tar.gz && \
tar -xz --strip-components=1 -f v1.4.0.tar.gz && \
rm -f v1.4.0.tar.gz && \
./configure --prefix=/usr --enable-sasl && \
make -j && \
make install |
|
@blaketastic2 I'm trying to build with yours, and I'm getting multiple errors:
Full dump: |
|
Yea, I’m not able to get 1.4.0 to build. I have it working for 1.3.0 and that can be seen here #461 |
|
@blaketastic2 @fals Set |
agalue
added a commit
to agalue/opennms-drift-kubernetes
that referenced
this issue
Apr 19, 2020
Some changes required on Dockerfile to avoid: confluentinc/confluent-kafka-go#454
|
@bschofield Building with |
|
@blaketastic2 Your broker fail message sounds like it's caused by the included librdkafka not having the SSL support you need. Try compiling your own librdkafka (1.4.0) and then building the go app with |
|
I ended up getting past this by continuing to copy the ca-certificates into the docker image as I mentioned earlier, and then making sure the consumer set:
Setting this prior to 1.4.0 was not needed, and instead was able to find the ssl ca on its own. |
|
I am getting: when building: |
|
Ok seems like this works: e.g. I think the key points:
|
|
Hi, Got this issue too, added the But a question, now my Jenkins builds take twice the previous build time... Anything that could explain that? To be honest, due to that I'm wondering if I'm not gonna stay with embedding librd on my own 😭 |
|
The |
|
@edenhill I have 3 steps:
Whereas previously before the Confluent update and
The detailed commands are the following ones:
The difference is quite significative 🤔 |
|
That does seem odd, maybe you can pass -x to the build to get an idea what steps are taking longer with the static builds. |
|
@edenhill , after multiple retries of building the same Git commit, I succeeded lowering So it could come from the build servers... A bit weird but better to consider this than a big change with embedded Kafka lib. Thank you for answering 👍 , |
|
After following @lifeofguenter I was able to build but not run the file. FROM golang:1.14-alpine AS builder
RUN set -ex &&\
apk add --no-progress --no-cache \
gcc \
musl-dev
# Set necessary environmet variables needed for our image
ENV GO111MODULE=on \
# CGO_ENABLED=0 \
GOOS=linux \
GOARCH=amd64 \
# Move to working directory /build
WORKDIR /build
# Copy and download dependency using go mod
COPY go.mod .
COPY go.sum .
RUN go mod download
# Copy the code into the container
COPY . .
# Build the application
RUN go get -d -v
RUN go build -a -tags musl -o main myproject.go
# Move to /dist directory as the place for resulting binary folder
WORKDIR /dist
# Copy binary from build to main folder
RUN cp /build/main .
# Build a small image
FROM scratch
COPY --from=builder /dist/main /
# Command to run
ENTRYPOINT ["/main"]I get the following error on running the file. standard_init_linux.go:211: exec user process caused "no such file or directory" |
|
Fixed by creating a static link RUN go build -a -tags musl -installsuffix cgo -ldflags '-extldflags "-static"' -o main myproject.go |
|
So I get the error mentioned in the description while running tests for my project via jenkins. (not sure if this is related) Dockerfile Jenkins snippet |
|
Can't reproduce with vanilla docker image: $ docker run -it golang:1.14-alpine /bin/sh
Unable to find image 'golang:1.14-alpine' locally
1.14-alpine: Pulling from library/golang
df20fa9351a1: Already exists
ed8968b2872e: Already exists
a92cc7c5fd73: Already exists
f17c8f1adafb: Pull complete
03954754c53a: Pull complete
Digest: sha256:e9f6373299678506eaa6e632d5a8d7978209c430aa96c785e5edcb1eebf4885e
Status: Downloaded newer image for golang:1.14-alpine
/go # apk add gcc musl-dev make git
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/x86_64/APKINDEX.tar.gz
(1/18) Installing libgcc (9.3.0-r2)
(2/18) Installing libstdc++ (9.3.0-r2)
(3/18) Installing binutils (2.34-r1)
(4/18) Installing gmp (6.2.0-r0)
(5/18) Installing isl (0.18-r0)
(6/18) Installing libgomp (9.3.0-r2)
(7/18) Installing libatomic (9.3.0-r2)
(8/18) Installing libgphobos (9.3.0-r2)
(9/18) Installing mpfr4 (4.0.2-r4)
(10/18) Installing mpc1 (1.1.0-r1)
(11/18) Installing gcc (9.3.0-r2)
(12/18) Installing nghttp2-libs (1.41.0-r0)
(13/18) Installing libcurl (7.69.1-r0)
(14/18) Installing expat (2.2.9-r1)
(15/18) Installing pcre2 (10.35-r0)
(16/18) Installing git (2.26.2-r0)
(17/18) Installing make (4.3-r0)
(18/18) Installing musl-dev (1.1.24-r9)
Executing busybox-1.31.1-r16.trigger
OK: 157 MiB in 33 packages
/go # cd src/
/go/src # mkdir fo
/go/src # cd fo
/go/src/fo # cat > main.go
package main
import (
"github.com/confluentinc/confluent-kafka-go/kafka"
"fmt"
)
func main () {
_, vs := kafka.LibraryVersion()
fmt.Printf("Kafka version %s\n", vs)
}
/go/src/fo # go get -tags musl
/go/src/fo # go build -tags musl
/go/src/fo # ./fo
Kafka version 1.4.2
/go/src/fo # ldd fo
/lib/ld-musl-x86_64.so.1 (0x7f4f65cb7000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f4f65cb7000)
/go/src/fo # |
|
So the build works. When I run tests on it is when I get the error. i.e on running |
|
You might need |
|
I am facing the same issue, from the container I can run what I am missing ? |
works, for sure on alpine, even with CGO_ENABLED=1 |
|
I struggled with this, and didn't realize we need the |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
When using the 1.4.0 release, the bundled librdkafka fails to work as intended with the latest alpine.
How to reproduce
Build an application which uses github.com/confluentinc/confluent-kafka-go, in a container, using the alpine:3.11 image (which, at the time of writing, is actually alpine:3.11.5) as a base. Errors such as the following will be displayed:
Installing a fresh build of librdkafka 1.4.0 into the container and building the application with
-tags dynamicresolves the issue.Could possibly be because the bundled librdkafka was built from a :edge release of alpine, instead of :latest?
Checklist
Please provide the following information:
LibraryVersion()): 1.4.0, bundled."debug": ".."as necessary): N/AThe text was updated successfully, but these errors were encountered: