[Bug]: renew not working with reverse-proxy and mtls=false

[juju4]

Steps to Reproduce

I setup a certificate server (step 0.28.2 on ubuntu 24.04.1) and a nginx reverse proxy.
Issuing certificate works fine from step service or nginx.
But renewal does not work when using nginx ca-url (port 443). It works if accessing directly step service (port 8443).
"mtls=false" was used. "--mtls false" as described in docs did not work with step-cli for me.
It does not seem there is any debug/verbose option to check where getting invalid character which is probably the start of a html page.

From cert-renewer systemd unit

ExecStart=/usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem --mtls false --force ${CERT_LOCATION} ${KEY_LOCATION} (code=exited, status=1/FAILURE)
# results in
Jan 08 21:50:43 myhost.internal step[3935]: too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'

Manual testing

root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem --mtls false /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'
root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value
root@myhost:~# /usr/bin/step ca renew --mtls false --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
too many positional arguments were provided in 'step ca renew <crt-file> <key-file>'
root@myhost:~# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem /etc/ssl/certs/myhost.crt /etc/ssl/private/myhost.key
error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value

Thanks

Your Environment

• OS - Ubuntu 24.04.1
• step CLI Version - 0.28.2

Expected Behavior

Renewal to work

Actual Behavior

Renewal fails

Additional Context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[hslatman]

Hey @juju4,

Can you try --mtls=false in your manual testing (and cert-renewer systemd unit)? I believe the --mtls false gets interpreted differently than you expect.

As for the < character: it's possible that your proxy is serving an error page when trying to upstream to the CA server. I suppose because those invocations don't have --mtls=false, they actually are terminated by the mTLS endpoint, and a TLS error is returned. So I think if you provide --mtls=false, you won't get that error message. Alternatively, you could try inspecting the HTML in a browser.

[hslatman]

One location where --mtls false was being mentioned was updated in this PR: smallstep/docs#374.

[juju4]

I did use mtls=false too. forgot to include in the list and it returns same error "error renewing certificate: failed decoding CA error response: invalid character '<' looking for beginning of value".

I would want to debug with curl as /renew needs a POST but I don't know the syntax needed. Browser /renew returns blank page with HTTP 405 status code. while on above server logs a 400 status code.

# curl -X POST https://certs.internal/renew
{"status":400,"message":"The request could not be completed: missing client certificate."}
# curl -X POST https://certs.internal/renew -d @/etc/ssl/certs/myhost.crt 
{"status":400,"message":"The request could not be completed: missing client certificate."}

[hslatman]

Can you try it with GODEBUG=http2debug=2 step ca renew ...? This will output the HTTP communication, and should let you inspect the response.

It's possible to use curl, but you would need to obtain a token first, and that'll involve a few more steps.

[juju4]

# journalctl -u cert-renewer@nginx -l --since yesterday
[...]
Jan 24 13:32:18 MYHOST step[50147]: certificate does not need renewal
Jan 24 13:32:18 MYHOST systemd[1]: [email protected]: Skipped due to 'exec-condition'.
Jan 24 13:32:18 MYHOST systemd[1]: Condition check resulted in [email protected] - Certificate renewer for nginx being skipped.
Jan 24 13:49:10 MYHOST systemd[1]: Starting [email protected] - Certificate renewer for nginx...
Jan 24 13:49:10 MYHOST step[58981]: failed decoding CA error response: invalid character '<' looking for beginning of value
Jan 24 13:49:10 MYHOST step[58981]: error renewing certificate
Jan 24 13:49:10 MYHOST step[58981]: github.com/smallstep/cli/command/ca.(*renewer).Renew
Jan 24 13:49:10 MYHOST step[58981]:         github.com/smallstep/cli/command/ca/renew.go:474
Jan 24 13:49:10 MYHOST step[58981]: github.com/smallstep/cli/command/ca.renewCertificateAction
Jan 24 13:49:10 MYHOST step[58981]:         github.com/smallstep/cli/command/ca/renew.go:331
Jan 24 13:49:10 MYHOST step[58981]: github.com/smallstep/cli/command/ca.renewCertificateCommand.ActionFunc.func1
Jan 24 13:49:10 MYHOST step[58981]:         github.com/smallstep/[email protected]/command/command.go:38
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.HandleAction
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/app.go:522
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.Command.Run
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/command.go:175
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.(*App).RunAsSubcommand
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/app.go:405
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.Command.startApp
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/command.go:380
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.Command.Run
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/command.go:103
Jan 24 13:49:10 MYHOST step[58981]: github.com/urfave/cli.(*App).Run
Jan 24 13:49:10 MYHOST step[58981]:         github.com/urfave/[email protected]/app.go:277
Jan 24 13:49:10 MYHOST step[58981]: main.main
Jan 24 13:49:10 MYHOST step[58981]:         ./main.go:73
Jan 24 13:49:10 MYHOST step[58981]: runtime.main
Jan 24 13:49:10 MYHOST step[58981]:         runtime/proc.go:272
Jan 24 13:49:10 MYHOST step[58981]: runtime.goexit
Jan 24 13:49:10 MYHOST step[58981]:         runtime/asm_amd64.s:1700
Jan 24 13:49:10 MYHOST systemd[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Jan 24 13:49:10 MYHOST systemd[1]: [email protected]: Failed with result 'exit-code'.
Jan 24 13:49:10 MYHOST systemd[1]: Failed to start [email protected] - Certificate renewer for nginx.
Jan 24 14:03:05 MYHOST systemd[1]: Starting [email protected] - Certificate renewer for nginx...
Jan 24 14:03:06 MYHOST step[3035]: failed decoding CA error response: invalid character '<' looking for beginning of value
Jan 24 14:03:06 MYHOST step[3035]: error renewing certificate
Jan 24 14:03:06 MYHOST step[3035]: github.com/smallstep/cli/command/ca.(*renewer).Renew
[...]
# cat /etc/systemd/system/cert-renewer\@nginx.service.d/override.conf 
[Service]
; `Environment=` overrides are applied per environment variable. This line does not
; affect any other variables set in the service template.
Environment=CERT_LOCATION=/etc/ssl/certs/MYHOST.crt \
            KEY_LOCATION=/etc/ssl/private/MYHOST.key \
            STEPDEBUG=1 \
            GODEBUG=http2debug=2

WorkingDirectory=/etc/ssl

; Restart service after the certificate is successfully renewed.
ExecStartPost=/usr/bin/systemctl restart nginx.service

even with the extra trace, I don't see where error comes from.

[hslatman]

Can you try that with the manual invocation instead?

[juju4]

same

# export CERT_LOCATION=/etc/ssl/certs/MYHOST.internal.crt KEY_LOCATION=/etc/ssl/private/MYHOST.internal.key STEPDEBUG=1 GODEBUG=http2debug=2
# /usr/bin/step ca renew --ca-url=https://certs.internal --root=/usr/share/ca-certificates/stepca-internal-roots.pem --mtls=false --force  ${CERT_LOCATION} ${KEY_LOCATION}
failed decoding CA error response: invalid character '<' looking for beginning of value
error renewing certificate
github.com/smallstep/cli/command/ca.(*renewer).Renew
	github.com/smallstep/cli/command/ca/renew.go:474
github.com/smallstep/cli/command/ca.renewCertificateAction
	github.com/smallstep/cli/command/ca/renew.go:331
github.com/smallstep/cli/command/ca.renewCertificateCommand.ActionFunc.func1
	github.com/smallstep/[email protected]/command/command.go:38
github.com/urfave/cli.HandleAction
	github.com/urfave/[email protected]/app.go:522
github.com/urfave/cli.Command.Run
	github.com/urfave/[email protected]/command.go:175
github.com/urfave/cli.(*App).RunAsSubcommand
	github.com/urfave/[email protected]/app.go:405
github.com/urfave/cli.Command.startApp
	github.com/urfave/[email protected]/command.go:380
github.com/urfave/cli.Command.Run
	github.com/urfave/[email protected]/command.go:103
github.com/urfave/cli.(*App).Run
	github.com/urfave/[email protected]/app.go:277
main.main
	./main.go:73
runtime.main
	runtime/proc.go:272
runtime.goexit
	runtime/asm_amd64.s:1700

[hslatman]

Interesting. Seeing the same behavior, specifically, no HTTP debug output using GODEBUG. Haven't found out why yet, though. Maybe it doesn't try HTTP2 at all.

I think for the test you could maybe just try a POST request to /renew using curl.

[juju4]

# curl -X POST https://certs.internal/renew
{"status":400,"message":"The request could not be completed: missing client certificate."}
# curl -X POST https://certs.internal/renew -d @/etc/ssl/certs/myhost.crt 
{"status":400,"message":"The request could not be completed: missing client certificate."}

how different from those?

[hslatman]

Try it with curl -H "Authorization: Bearer invalid" -X POST https://certs.internal/renew

[juju4]

# curl -H "Authorization: Bearer invalid" -X POST https://certs.internal/renew
{"status":401,"message":"error validating renew token"}