Can't run the v3.27.2 release on Debian because of glibc issue

[DavidZidar]

It seems that the version 3.27.2 dnscontrol requires a newer glibc-library than what seems to available on Debian.

Here is the error that I'm getting:

./dnscontrol: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./dnscontrol)
./dnscontrol: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./dnscontrol)

My system is up to date and seems to have version 2.31-13+deb11u5 of glibc installed.

Version 3.27.1 runs just fine so something has happened between these two releases.

[DavidZidar]

I can add that I'm trying to run the dnscontrol-Linux-binary from the GitHub releases page, but I've also downloaded the .deb-package and tried installing that and get the same issue.

[DavidZidar]

I tried looking at the diff but I don't see any relevant changes that I can tell but I don't know Go.

It seems though, when I look in your GitHub Actions logs, that dnscontrol v3.27.1 was built using Go 1.19.6 and dnscontrol v3.27.2 was built using Go 1.20.1. Could that be the issue?

[tlimoncelli]

(sorry for the delayed reply. I'm traveling and have limited internet access)

I'm getting the same error.

I've set 3.27.1 as the "latest release" and marked 3.27.2 as a "pre-release". Hopefully that will steer people away from the bad release until I can look into this further.

Tom

CC @cdhunt for visibility.

[DavidZidar]

I reported this less than 2 hours ago, replying that fast is plenty quick enough. 😄

Thanks for looking into this.

[cdhunt]

Build environments:

• 3.72.1:
  • Kernel Version: 5.15.0-1028-aws, Operating System: Ubuntu 20.04.5 LTS
  • using image cimg/go@sha256:85a5daa2c826faa566f1cb0d56e7b9bc6a378e641ed1a8254440e198e8dfe9b9
• 3.27.2:
  • Kernel Version: 5.15.0-1030-aws, Operating System: Ubuntu 20.04.5 LTS
  • using image cimg/go@sha256:85a5daa2c826faa566f1cb0d56e7b9bc6a378e641ed1a8254440e198e8dfe9b9

[cdhunt]

https://ubuntu.com/security/notices/USN-5876-1

[DavidZidar]

Go 1.20 release notes mention something about linker changes, might be relevant.

On Linux, the linker now selects the dynamic interpreter for glibc or musl at link time.

https://tip.golang.org/doc/go1.20

[DavidZidar]

Might be the same as this issue some people run into.
golang/go#58550

[tlimoncelli]

It looks like #2141 fixes this problem, but I don't know if it is the best way to fix it. Either way, let's keep it for now and kick the can down the road until otherwise.

I have a very busy day but hopefully tomorrow I can do a release and verify that the problem is gone. I can test centos but not debian, btw.

[DavidZidar]

I can test Debian 11 if I just get a binary somehow.

[cdhunt]

Here are all the binaries from the test build.

https://app.circleci.com/pipelines/github/StackExchange/dnscontrol/2662/workflows/1a7c0909-18d4-499d-a37c-d27f7ff1d507/jobs/16883/artifacts

[DavidZidar]

I tested the dist/build_linux_amd64_v1/dnscontrol binary and it seems to work fine.

Things that I tried that didn't work before:

• Printed the help page
• Wrote the types file
• Checked my configuration

[tlimoncelli]

@DavidZidar thanks for the testing! The next release should fix this.

[DavidZidar]

Hi again, I'm sad to say that v3.28.0 did not fix this issue even though it says so in the release highlights.

[DavidZidar]

The reason the CircleCI build was working is probably because that build was using Go 1.18, I noticed that now that I dug in the logs. The actual released version was built using Go 1.20.2, so it looks like the changes in #2141 didn't resolve the issue.

[tlimoncelli]

Hi again, I'm sad to say that v3.28.0 did not fix this issue even though it says so in the release highlights.

Shit. I apologize for not testing it more fully. I saw that it worked in #2141 and assumed that would carry through to the release.

I'm at a loss for how to fix this. Would you mind taking a look?

[DavidZidar]

I'm a .NET developer so I'm out of my depth with the Go stuff, but I'm skimming the issues in the Go repository on GitHub and I found this interesting comment:

The workarounds I can see for your use-case are:

• Build the program on the system with the older glibc (so that it uses the older symbols).
• Set CGO_ENABLED=0 to build a binary that doesn't depend on glibc at all.
• Set various flags (see cmd/go: build: add -static flag #26492) to build a fully-static cgo-enabled binary.

So one (temporary?) solution might be to change the build configuration to use an older runner like ubuntu-20.04 instead of ubuntu-latest so that older libraries are used.

A better solution might be to set CGO_ENABLED=0 using an environment variable as it's apparently enabled by default. From what I'm reading that should create static binary with no dependencies that should work. I'm assuming that will make it larger as a trade-off.

[DavidZidar]

The comment before the one I linked is also interesting, it might actually link to the root cause of the issue. It seems they stopped shipping .a-files with Go in version 1.20 to make the Go distribution smaller which means that all of a sudden it matters what version of glibc is installed on the system doing the building. So the combination of Go 1.20 plus ubuntu-latest which is now Ubuntu 22.04 which contains newer versions of glibc makes it so that the binary no longer works on systems with older versions of glibc.

Setting CGO_ENABLED=0 to false should solve this, but I don't know how large the binary will get.

[tlimoncelli]

I agree with your conclusion. Sadly I can't figure out how to get goreleaser to set CGO_ENABLED=0 :-(

I'll give it another try this weekend. If not, I'll be at SRECon next week and maybe I'll find someone that can help there :)

[cdhunt]

We're doing it just like it's documented here.

    # Default: `os.Environ()` merged with what you set the root `env` section.
    env:
      - CGO_ENABLED=0

[cdhunt]

The v3.27.2 release doesn't include the -linkmode=internal fix.

[cdhunt]

v3.28.0 does include the fix. Does that release work for you, @DavidZidar?

[DavidZidar]

@cdhunt That's the version I tried today and it's not working.

I notice that you've configured CGO_ENABLED=0 in .goreleaser.yml but the GitHub Action isn't actually using GoReleaser at all. The CircleCI build is though. So if the GitHub Action is the one that is producing the releases then the CGO_ENABLED-flag isn't really configured.

I'm thinking if you modify release.yml like this it might work?

    - name: Build binaries
      run: go run build/build.go
      env:
        CGO_ENABLED: 0

[DavidZidar]

Or maybe add a Setenv call in build.go.

[tlimoncelli]

Oh! D'oh! Yes, the change needs to be in release.yaml, I was changing the old file.

Please take a look at release v3.29.0-test2 and tell me if that works.

[DavidZidar]

@tlimoncelli Looks like the build failed because it couldn't upload the RPM package.

The Bundle RPM step had this warning:

Package version '3.29.0-test2' includes dashes, converting to underscores

So the file that was created had an underscore in the tag name dnscontrol-3.29.0_test2-1.x86_64.rpm, then it tried to upload a file with a dash dnscontrol-3.29.0-test2-1.x86_64.rpm which of course didn't exist.

[tlimoncelli]

Can someone test the Debian binaries on https://github.com/StackExchange/dnscontrol/releases/tag/untagged-28f9b2f50edfd865d530 please?

I tested the Linux binary on CentOS 7 and the FreeBSD binary on FreeBSD 13.1.

[DavidZidar]

I would help but I'm getting a 404 on that link.

[tlimoncelli]

Ah! I didn't realize that only admins can see draft releases. I've released it as a "pre-release". Hope that helps!

https://github.com/StackExchange/dnscontrol/releases/tag/v3.29.0

[DavidZidar]

That worked better and it's working perfectly in my Debian 11 install too. 😄

[DavidZidar]

I did discover a separate problem with the types that's unrelated to glibc and reported that as issue #2213.

[tlimoncelli]

That worked better and it's working perfectly in my Debian 11 install too. 😄

Glad to hear it! Thank you for your patience.

[valodzka]

I tried 3.29.1 on debian 11 and got same issue (3.29.0 also not working):

/opt/dnscontrol-v3.29.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /opt/dnscontrol-v3.29.1)
/opt/dnscontrol-v3.29.1: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /opt/dnscontrol-v3.29.1)

[DavidZidar]

@tlimoncelli I'm actually having the same issue as @valodzka with the dnscontrol-Linux binary, that one wasn't uploaded when I did my tests earlier so I couldn't test it.

Somehow the binary in the dnscontrol_3.29.1_linux_amd64.tar.gz archive works but not the non-archived file. The file sizes are even different which is strange. Shouldn't it be the exact same binary in both cases?

[valodzka]

Can confirm, binary from archive is working, from dnscontrol-Linux isn't:

user$ ls -l dnscontrol*
-rwxr-xr-x 1 user group 33570816 Mar 24 16:12 dnscontrol
-rw-r--r-- 1 user group 33779712 Mar 24 17:00 dnscontrol-Linux

user$ ldd dnscontrol-Linux
./dnscontrol-Linux: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./dnscontrol-Linux)
./dnscontrol-Linux: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./dnscontrol-Linux)
        linux-vdso.so.1 (0x00007ffd0097d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f81bd17e000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f81bd35d000)

user$ ldd dnscontrol
        not a dynamic executable

[tlimoncelli]

Re-opening.

[tlimoncelli]

Fixed in 5d8cd9f

I hope. Damn, this is depressing.

[DavidZidar]

Seems to work as expected in v3.30.0. May I ask why the fix was reverted in the first place?

[tlimoncelli]

I'm glad it works now!

Sorry this took so long. It wasn't just blocking you... internally we reverted to an older binary and were blocked on using one of the new features as a result.

Why was the fix reverted? In all honesty it was a series of errors, all my fault. I'm not sure if the fix was ever merged to master or maybe it was merged but reverted as part of other revert. The final fix was realizing that I was updating the wrong process: we're in the middle of changing between GHA and CircleCI and I think I updated the pipeline that is NOT in use. There was at least one point where I checked the binary on a CentOS system (and it worked), but somehow the next official release didn't have the fix. I can only conclude whatever I did to test it was invalid.

[DavidZidar]

I see, so it was just a simple mistake then. The fix was there in master and I verified that it was, but then it was mysteriously reverted with no explanation and the next release was broken again. So I was just surprised to see that when it was fixed again the fix was exactly the same as the first time. 😅

Works well for me now in any case, should I close the issue?

[tlimoncelli]

Thanks for understanding. I'll close the issue.