RPC Error

[dtonnesen]

How can the Team help you today?

Attempting to install CSM. Vanilla K8s, v1.20.9 Ubuntu. Using default parameters and only modifying the 4 parameters in values.yaml with no certificates.

root@dsib0211:~/csm# more values.yaml

jwtKey: key

cipherKey: "aasdfgafhgshsffadgshsdffgsdggggg"

adminUserName: admin

adminPassword: admin

helm install -n csm-installer --set-string scheme=http --set-string dbSSLEnabled="false" --create-namespace csm-installer dell/csm-installer -f values.yaml

NAME: csm-installer
LAST DEPLOYED: Tue Nov 16 10:15:21 2021
NAMESPACE: csm-installer
STATUS: deployed
REVISION: 1
TEST SUITE: None

Deployment seems to succeed but the cockroachdb does not start:
NAMESPACE NAME READY STATUS
csm-installer cluster-init-sjtdr 1/1 Running
csm-installer cockroachdb-0 0/1 Running
csm-installer cockroachdb-1 0/1 Running
csm-installer dell-csm-installer-86665ffb7d-dphzh 1/1 Running

Looking at the logs for the cluster-init I see constant repetition of error below which I assume is referring to the db:

kubectl logs -f cluster-init-sjtdr -n csm-installer

warning: node not ready to perform cluster initialization: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" (retrying)

cockroachdb logs have connectivity issue:

kubectl logs -f cockroachdb-0 -n csm-installer

W211116 15:16:03.965422 115 vendor/google.golang.org/grpc/internal/channelz/logging.go:73 ⋮ [-] 26 ‹grpc: addrConn.createTransport failed to connect to {cockroachdb-1.cockroachdb:26257 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...›
W211116 15:16:03.965681 46 server/init.go:374 ⋮ [n?] 27 outgoing join rpc to ‹cockroachdb-1.cockroachdb:26257› unsuccessful: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"›

I have thoroughly searched for troubleshooting information and tried changing values like host and api server in the values.yaml but same error. I'm happy to provide additional information if useful. Thanks.

[hoppea2]

Thanks for you submission @dtonnesen, we'll triage this shortly.

[tdawe]

Hi @dtonnesen , can you re-install the CSM installer and set dbInstanceCount to 1 in the values.yaml file. This will only deploy a single instance of cockroachdb and may help get around the issue where multiple instances are unable to join.

[dtonnesen]

Thank you I tried that too I should have mentioned sorry. I can do it again if you think the logs might be different.

[tdawe]

Yes, please try again and check if the logs are different. Prior to re-installing you can delete the /var/lib/cockroachdb directories from the k8s worker nodes (see https://dell.github.io/csm-docs/docs/deployment/troubleshooting/#why-does-the-cluster-init-pod-show-the-error-cluster-has-already-been-initialized) which are used to persist the database data. This will ensure it's a fresh install and not using any old data from cockroachdb.

[dtonnesen]

Sure, I did do that last time also but I'll do it again.

[dtonnesen]

Looks to be the same errors:

I211117 13:12:51.808172 28 server/init.go:197 ⋮ [n?] 30 awaiting cockroach init or join with an already initialized node
W211117 13:13:11.808789 95 vendor/google.golang.org/grpc/internal/channelz/logging.go:73 ⋮ [-] 31 ‹grpc: addrConn.createTransport failed to connect to {cockroachdb-0.cockroachdb:26257 0 }. Err: connection error: desc = "transport: Error while dialing dial tcp: i/o timeout". Reconnecting...›
W211117 13:13:11.809074 93 server/init.go:374 ⋮ [n?] 32 outgoing join rpc to ‹cockroachdb-0.cockroachdb:26257› unsuccessful: ‹rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"›

transport: Error while dialing dial tcp: i/o timeout" (retrying)
warning: node not ready to perform cluster initialization: initial connection heartbeat failed: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout" (retrying)

[dtonnesen]

@tdawe @hoppea2 Any additional thoughts on what I can try here? I've done all I can outside of the CSM installation, including implementing the replication module, so this is impeding my ability to finish the project. Thanks.

[tdawe]

Currently reviewing the environment

[tdawe]

Closing this question as we've discussed a workaround using the helm charts.