Cannot set DNS server in Docker image

[tindzk]

After upgrading to ghcr.io/letsencrypt/pebble:2.5.1, I am seeing the following error in our test suite:

INFO event=SendRequest url=https://localhost:14000/authZ/WKrIeCnHQULq6mp3BD0eEYTlwq7M_MInYnISVm5RmXA
INFO event=ReceivedResponse location=None body="{\n   \"status\": \"invalid\",\n   \"identifier\": {\n      \"type\": \"dns\",\n      \"value\": \"<redacted>\"\n   },\n   \"challenges\": [\n      {\n         \"type\": \"tls-alpn-01\",\n         \"url\": \"https://localhost:14000/chalZ/qCiMNJz2zm1mLtYTUn9hXC4cizqg25g04UI1B2AWrjk\",\n         \"token\": \"TlK9my79nS0tSjtdmugDJHWUzE6N1MdK_nMjKQrd0zU\",\n         \"status\": \"invalid\",\n         \"validated\": \"2024-03-19T16:51:56Z\",\n         \"error\": {\n            \"type\": \"urn:ietf:params:acme:error:malformed\",\n            \"detail\": \"Error occurred while resolving URL \\\"<redacted>\\\": \\\"lookup <redacted> on 8.8.8.8:53: no such host\\\"\",\n            \"status\": 400\n         }\n      }\n   ],\n   \"expires\": \"2024-03-19T17:51:55Z\"\n}"

This suggests that the -dnsserver setting is not honoured. As a workaround, I was able to specify a custom DNS server for the entire Docker container, but this required changing the port to 53.

After further investigation, the issue seems to be that container parameters are being ignored altogether. The following invocation should fail, but the invalid parameter goes unnoticed:

docker run ghcr.io/letsencrypt/pebble:2.5.1 abc

The container listing shows that the command is translated into /app abc:

CONTAINER ID   IMAGE                              COMMAND                  CREATED         STATUS         PORTS                                         NAMES
e1b1b497e357   ghcr.io/letsencrypt/pebble:2.5.1   "/app abc"               2 minutes ago   Up 2 minutes                                                 recursing_kare

Another issue I found with the new Docker image is that it does not expose any ports, requiring users to override the ExposedPorts setting when using the Docker daemon directly. Previously, the Dockerfile contained these two lines:

EXPOSE 14000
EXPOSE 15000

[mcpherrinm]

Thanks, those seen like issues we should correct.

[shred]

EDIT: I'm sorry, I just realized this should rather be a new issue.

[philipahlberg]

As a workaround, I was able to specify a custom DNS server for the entire Docker container, but this required changing the port to 53.

Is there anything specific you had to do to make this work? I've tried adding dns: under pebble in compose.yml, and appending -dns01 :53 to the pebble-challtestsrv command, but it doesn't seem to work.

[tindzk]

As a workaround, I was able to specify a custom DNS server for the entire Docker container, but this required changing the port to 53.

Is there anything specific you had to do to make this work? I've tried adding dns: under pebble in compose.yml, and appending -dns01 :53 to the pebble-challtestsrv command, but it doesn't seem to work.

I have not tried pebble-challtestsrv as I am running a custom DNS server that listens on 172.17.0.1:53. Pebble is able to resolve domains when I start the container with --dns 172.17.0.1.

[mcpherrinm]

After further investigation, the issue seems to be that container parameters are being altogether. The following invocation should fail, but the invalid parameter goes unnoticed

This is a pebble bug, not a container one. Positional (non - flag arguments) are ignored by pebble and doesn't throw an error.

Providing a flag properly "fails":

 docker run ghcr.io/letsencrypt/pebble:2.5.1 -test
flag provided but not defined: -test

So it is not the source of -dnsserver not working for you

[mitar]

I can confirm. I spent two days debugging this now. :-( Docker image for pebble does not respect anymore CLI arguments. So it is not just -dnsserver, it ignores -config as well. Try running with -config /nonexistent and it will start. v2.3.1 does not start.

Furthermore, pebble-challtestsrv Docker image also ignores CLI arguments. E.g., -defaultIPv4 is ignored.

[mcpherrinm]

I'm sorry, but I can't replicate.

% docker run ghcr.io/letsencrypt/pebble:2.5.2 -config /nonexistent
Pebble 2024/05/24 15:16:03 Starting Pebble ACME server
Reading JSON config file into config structure: open /nonexistent: no such file or directory

You'll need to share specific steps to reproduce.

[pgporada]

Running challtestsrv with ports mapped:

phil@computer $ docker rm challtestsrv 2>&1; docker run -p 5001:5001 -p 5002:5002 -p 5003:5003 -p 8053:8053 -p 8055:8055 -p 8443:8443 --name challtestsrv ghcr.io/letsencrypt/pebble-challtestsrv:latest
challtestsrv
pebble-challtestsrv - 2024/05/24 15:29:42 Added DNS-01 TXT challenge for Host "_acme-challenge.atbdodijeib.com." - Value "67w4W3Zfelb0yOQGgv2xsjS_s4Bpf-qcAsfbb6V2ieQ"
pebble-challtestsrv - 2024/05/24 15:29:47 Removed DNS-01 TXT challenge for Host "_acme-challenge.atbdodijeib.com."

Running pebble in a container with ports mapped:

phil@computer $ CHALLTESTSRV=$(docker inspect challtestsrv | jq -r '.[].NetworkSettings.Networks.bridge.IPAddress')
phil@computer $ docker run -p 14000:14000 -p 15000:15000 ghcr.io/letsencrypt/pebble:2.5.1 -config ./test/config/pebble-config.json -dnsserver ${CHALLTESTSRV}:8053
Pebble 2024/05/24 15:29:42 Added order "acIYt-AAF6bNtaF2yx3XzShEqsNCNIfenSUWVFqyOTU" to the db
Pebble 2024/05/24 15:29:42 There are now 4 orders in the db
Pebble 2024/05/24 15:29:42 POST /authZ/ -> calling handler()
Pebble 2024/05/24 15:29:42 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:42 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"atbdodijeib.com"}, Challenge:(*core.Challenge)(0xc0002dcb40), Account:(*core.Account)(0xc000399980)}
Pebble 2024/05/24 15:29:42 Starting 3 validations.
Pebble 2024/05/24 15:29:42 Sleeping for 2s seconds before validating
Pebble 2024/05/24 15:29:42 Sleeping for 4s seconds before validating
Pebble 2024/05/24 15:29:42 Sleeping for 0s seconds before validating
Pebble 2024/05/24 15:29:43 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:43 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:44 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:44 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:45 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:46 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:46 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:46 authz CPCZQUmdHCY0xLwxy-LL4rWNzn-DY8-E6RDdLlfP_ks set VALID by completed challenge 9KTx5N06l2EHypr4Kqv3mWtiknNHPwS_-ifI6U61-yQ
Pebble 2024/05/24 15:29:47 POST /chalZ/ -> calling handler()
Pebble 2024/05/24 15:29:47 POST /my-order/ -> calling handler()
Pebble 2024/05/24 15:29:47 POST /finalize-order/ -> calling handler()
Pebble 2024/05/24 15:29:47 Order acIYt-AAF6bNtaF2yx3XzShEqsNCNIfenSUWVFqyOTU is fully authorized. Processing finalization
Pebble 2024/05/24 15:29:47 Issued certificate serial 644e59fdc6c4fe7d for order acIYt-AAF6bNtaF2yx3XzShEqsNCNIfenSUWVFqyOTU
Pebble 2024/05/24 15:29:52 POST /my-order/ -> calling handler()

Running an arbitrary ACME client test against pebble in a container succeeds.

phil@computer $ go test -test.run TestClient_FinalizeOrder
2024/05/24 11:29:42 using pebble directory at: https://localhost:14000/dir
PASS
ok      github.com/eggsampler/acme/v3   9.037s

You can see the random domain atbdodijeib.com being added to challtestsrv by pebble.

Edit: I just saw that the following 2 port map lines were removed from the original linux.Dockerfile and windows.Dockerfile in favor of Dockerfile.release from this patch. The docker-compose.yml does expose these ports, but you have to be using docker compose, not running arbitrary containers like I did above.

EXPOSE 14000
EXPOSE 15000

As for pebble-challtestsrv, flags appear to be being respected there. Take for example the output of running these various commands and noticing how they differ depending on the given flag.

$ docker rm challtestsrv 2>&1; docker run -p 5001:5001 -p 5002:5002 -p 5003:5003 -p 8053:8053 -p 8055:8055 -p 8443:8443 --name challtestsrv ghcr.io/letsencrypt/pebble-challtestsrv:latest -dns01 "" -https01 "" -http01 ""
challtestsrv
pebble-challtestsrv - 2024/05/24 15:47:53 Creating DoH server on :8443
pebble-challtestsrv - 2024/05/24 15:47:53 Creating TLS-ALPN-01 challenge server on :5001
pebble-challtestsrv - 2024/05/24 15:47:53 Starting challenge servers
pebble-challtestsrv - 2024/05/24 15:47:53 Starting management server on :8055
pebble-challtestsrv - 2024/05/24 15:47:53 open : no such file or directory

$ docker rm challtestsrv 2>&1; docker run -p 5001:5001 -p 5002:5002 -p 5003:5003 -p 8053:8053 -p 8055:8055 -p 8443:8443 --name challtestsrv ghcr.io/letsencrypt/pebble-challtestsrv:latest -dns01 ""
challtestsrv
pebble-challtestsrv - 2024/05/24 15:46:32 Creating HTTP-01 challenge server on :5002
pebble-challtestsrv - 2024/05/24 15:46:32 Creating HTTPS HTTP-01 challenge server on :5003
pebble-challtestsrv - 2024/05/24 15:46:32 Creating DoH server on :8443
pebble-challtestsrv - 2024/05/24 15:46:32 Creating TLS-ALPN-01 challenge server on :5001
pebble-challtestsrv - 2024/05/24 15:46:32 Starting challenge servers
pebble-challtestsrv - 2024/05/24 15:46:32 Starting management server on :8055
pebble-challtestsrv - 2024/05/24 15:46:32 open : no such file or directory

$ docker rm challtestsrv 2>&1; docker run -p 5001:5001 -p 5002:5002 -p 5003:5003 -p 8053:8053 -p 8055:8055 -p 8443:8443 --name challtestsrv ghcr.io/letsencrypt/pebble-challtestsrv:latest -defaultIPv4 6.6.6.1
challtestsrv
...
pebble-challtestsrv - 2024/05/24 15:48:37 Answering A queries with 6.6.6.1 by default

[mitar]

@mcpherrinm: I am running it as:

$ docker run ghcr.io/letsencrypt/pebble:2.5.1 pebble -config /nonexistent
Pebble 2024/05/27 07:25:56 Starting Pebble ACME server
Pebble 2024/05/27 07:25:56 Generated new root issuer CN=Pebble Root CA 6fee8d with serial 4d1d9d2e052490d5 and SKI 2cab45a8292a9d00c6ec1da8ad19b16f270f9f66
Pebble 2024/05/27 07:25:56 Generated new intermediate issuer CN=Pebble Intermediate CA 337d7e with serial 4829eda1bd03fffa and SKI 7a0e43b46fa5aac111a46d1c84e48dd849170c09
Pebble 2024/05/27 07:25:56 Generated issuance chain: Pebble Root CA 6fee8d -> Pebble Intermediate CA 337d7e
Pebble 2024/05/27 07:25:56 Using certificate validity period of 157766400 seconds
Pebble 2024/05/27 07:25:56 Using system DNS resolver for ACME challenges
Pebble 2024/05/27 07:25:56 Configured to reject 5% of good nonces
Pebble 2024/05/27 07:25:56 Configured to attempt authz reuse for each identifier 50% of the time
Pebble 2024/05/27 07:25:56 Configured to show 3 orders per page
Pebble 2024/05/27 07:25:56 Management interface listening on: 0.0.0.0:15000
Pebble 2024/05/27 07:25:56 Root CA certificate available at: https://0.0.0.0:15000/roots/0
Pebble 2024/05/27 07:25:56 Listening on: 0.0.0.0:14000
Pebble 2024/05/27 07:25:56 ACME directory available at: https://0.0.0.0:14000/dir

This is how README has it documented, so with pebble in the command.

[mcpherrinm]

Thanks, I see the problem now. I think we can both fix the Pebble CLI to error in that case, and also fix the README. In the meantime, you can remove the pebble from the docker run command.