Certificate issues in current master branch (HEAD@510ea94)

[echupriyanov]

I’ve tried to run swarm manager built from master branch (HEAD@510ea94)
And I see following certificate issues in logs:

INFO[0000] Listening for HTTP                            addr=0.0.0.0:4000 proto=tcp
INFO[0000] Leader Election: Cluster leadership lost
INFO[0000] Leader Election: Cluster leadership acquired
DEBU[0000] Watch triggered with 4 nodes                  discovery=etcd
DEBU[0000] Failed to validate pending node: An error occurred trying to connect: Get https://swarm-node-04a.cybertonica.aws:2376/info: x509: certificate is valid for swarm-node-04a.cybertonica.aws, not swarm-node-05a.cybertonica.aws  Addr=swarm-node-04a.cybertonica.aws:2376
DEBU[0000] Failed to validate pending node: An error occurred trying to connect: Get https://swarm-node-03a.cybertonica.aws:2376/info: x509: certificate is valid for swarm-node-03a.cybertonica.aws, not swarm-node-05a.cybertonica.aws  Addr=swarm-node-03a.cybertonica.aws:2376
DEBU[0000] Failed to validate pending node: An error occurred trying to connect: Get https://swarm-node-06a.cybertonica.aws:2376/info: x509: certificate is valid for swarm-node-06a.cybertonica.aws, not swarm-node-05a.cybertonica.aws  Addr=swarm-node-06a.cybertonica.aws:2376
DEBU[0000] Start monitoring events                       id=MGVW:TYQU:XFOD:3PAF:POTR:NEZ4:NTG4:JEMN:FH4U:THWW:FLLY:OXGR name=swarm-node-05a.cybertonica.aws
INFO[0000] Registered Engine swarm-node-05a.cybertonica.aws at swarm-node-05a.cybertonica.aws:2376

Same error appears in build from v1.2.0-rc1 tag.
Build from tag v1.1.3 has no such errors.

[jpetazzo]

Confirmed -- just ran into this exact same issue on my test clusters. I use Consul for discovery (not that it should matter).

My setup works with 1.1.3-rc2 (7e9c6bd) but not with master (a5cba29).

[abronan]

/cc @vieux @dongluochen

[diogomonica]

What I don't understand here is this:

Get https://swarm-node-**04a**.cybertonica.aws:2376/info: x509: certificate is valid for swarm-node-04a.cybertonica.aws, not swarm-node-05a.cybertonica.aws Addr=swarm-node-04a.cybertonica.aws:2376

I don't understand how that error log like can even show 05a, when we're clearly connecting to 04a.

[echupriyanov]

@diogomonica There is a TLSClientConfig *tls.Config structure, which contains TLS configuration used to connect to Docker engines. Part of this structure is ServerName field. That field is usually empty, which means we only check if certificate received from engine matches the url we use to connect to it. But if its not empty, we also check if engine'e hostname matches ServerName in provided TLS config.

Current implementation of engine-api explicitly sets ServerName field if its empty. And since the same TLS config structure is re-used to connect to all engines in cluster, following is happening:

1. Swarm master connects to the first engine in cluster (ServerName is empty). During the connection establishment, ServerName is set to that engine's hostname.
2. Swarm tries to connect to second (third and so on) engine, but ServerName is already set to hostname of the first engine, and connection fails.

I've pushed a PR (docker/engine-api#173) to remove that behavior.

[diogomonica]

That makes sense, thank you for the detail.

"But if its not empty, we also check if engine'e hostname matches ServerName in provided TLS config."

I don't think this is a "we also check", I think this is a "we only check". IIRC, if you provide a ServerName, the underlying TLS stack will only check against the provided ServerName, so it's not an additive check, it's a replacement check.

[echupriyanov]

I’ve updated PR with a fix on Swarm level.

[echupriyanov]

Should be fixed by docker/engine-api#179

[dongluochen]

We will merge the change docker/engine-api#179. Thanks @echupriyanov!

[vieux]

fixed in #2065