dep: update etcd 3.2.16, kubernetes 1.10.0, k8s codegen 1.10

[hongchaodeng]

No description provided.

[hongchaodeng]

@etcd-bot retest this please

Jenkins config issue.

[raoofm]

v1.12.41 is out

[raoofm]

v3.2.11 is out

[raoofm]

v3.2.11

[raoofm]

v1.12.41

[hongchaodeng]

Two changes here:

• update the dep: etcd 3.2 and k8s 1.10 are aligned in deps so this would be easier to bump up together
• in k8s 1.10 the protocol of generated code has changed so I need to build another container gcr.io/coreos-k8s-scale-testing/codegen:1.10 and update it here.

[hongchaodeng]

There is some TLS issue.
etcd server log:

WARNING: 2018/03/05 23:20:11 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.

Might be relevant: etcd-io/etcd#8603

[hongchaodeng]

Tried with etcd 3.3.1 with more and better error message:

2018-03-06 00:09:53.475001 I | embed: rejected connection from "127.0.0.1:44500" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2018/03/06 00:09:53 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
2018-03-06 00:09:53.730840 I | embed: rejected connection from "10.28.2.119:55094" (error "remote error: tls: internal error", ServerName "example-d9hvr87bh4.example.default.svc")
2018-03-06 00:10:06.742990 I | embed: rejected connection from "10.28.2.119:55120" (error "remote error: tls: internal error", ServerName "example-d9hvr87bh4.example.default.svc")

Operator failed creating etcd client:

creating etcd client failed: context deadline exceeded

[hongchaodeng]

Update:
Seems like it was some dep mistake that dep 0.4 has changed the semantics: https://golang.github.io/dep/docs/Gopkg.toml.html#version

If so we also need to update test infra to bump up to the 0.4 dep. done

[hongchaodeng]

I have pinned down grpc version in Gopkg.toml :

[[override]]
  name = "google.golang.org/grpc"
  version = "=v1.7.5"

Otherwise it will somehow bump to "1.10.0" .

But I'm still hitting the TLS issue:

2018-03-06 05:26:12.671477 I | embed: rejected connection from "127.0.0.1:52016" (error "tls: failed to verify client's certificate: x509: certificate specifies an incompatible key usage", ServerName "")
WARNING: 2018/03/06 05:26:12 Failed to dial 0.0.0.0:2379: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"; please retry.
2018-03-06 05:26:15.048265 I | embed: rejected connection from "10.28.4.252:33988" (error "remote error: tls: internal error", ServerName "example-mx55hzcd7h.example.default.svc")

[hongchaodeng]

We dig into the TLS handshake process.

Here's some background: server will request certificate from client and verify the key usage of the cert:
https://github.com/golang/go/blob/6732fcc06df713fc737cee5c5860bad87599bc6d/src/crypto/tls/handshake_server.go#L731

We found that after bumping the client version as in this PR, the cert.ExtKeyUsage becomes different.
Previously it was ClientAuth, afterwards it becomes ServerAuth. The KeyUsage to verify is ClientAuth -- that's why the Verify() failed:
https://github.com/golang/go/blob/6732fcc06df713fc737cee5c5860bad87599bc6d/src/crypto/x509/verify.go#L340

And printed out

failed to verify client's certificate: x509: certificate specifies an incompatible key usage

[hongchaodeng]

After digging, the incompatible key usage and tls: bad certificate do not stop etcd from serving client requests.

When operator tried to create etcd client and failed, etcd logs tls: internal error:

2018-03-07 23:26:29.598194 I | embed: rejected connection from "10.28.2.219:55624" (error "remote error: tls: internal error", ServerName "example-q7ncrlm48d.example.default.svc")

[hongchaodeng]

OK. After I directly inject certs onto etcd operator volumes and creating TLSConfig without reading secrets, it works. I guess this is the direction to go. I will debug reading secrets.

[hongchaodeng]

I have made it work.

The issue is caused by previously we deleted the certs directory after creating TLSConfig. But it seems like 3.2.16 etcd client package needs to reload those files again. As a result after removing the code that deletes the certs dir, it works.

[hongchaodeng]

More details on the code:

TLSConfig has a lazy func GetClientCertificate():

https://github.com/coreos/etcd/blob/a537163e9e67a145035c3f6611788a6746044dfc/pkg/transport/listener.go#L180-L181

GetClientCertificate() doc:
https://github.com/golang/go/blob/c8aec4095e089ff6ac50d18e97c3f46561f14f48/src/crypto/tls/common.go#L372-L385

is called when a server requests a certificate from a client.

When it is called, it loads cert files from disk:

tlsutil.NewCert(info.CertFile, info.KeyFile, info.parseFunc)

Thus, we can't delete cert files.

[hasbro17]

@hongchaodeng SGTM. Do we need to update the default etcd base image to 3.2.16 as well?

[hongchaodeng]

Do we need to update the default etcd base image to 3.2.16 as well?

That would be ideal.
Let's change the client first and then the server too.

[hongchaodeng]

@hasbro17
Can you merge this?