cf dev is stuck on 'waiting for services to start'

[gmanoj79]

I was using pcfdev last week without any issues but today I am not able to start cf dev. It gets stuck at 'waiting for services to start'. I tried starting it with a .ova file that I downloaded but same issue.

cf dev start
Allocating 4096 MB out of 16384 MB total system memory (6462 MB free).
Importing VM...
Starting VM...
Provisioning VM...
Waiting for services to start...

I did a 'cf dev ssh' and here's the output of some commands (looks like 'consul_agent' is having certificate issues)

tail -f /var/vcap/monit/monit.log
vcap@agent-id-pcfdev-0:/var/vcap/monit$ sudo tail -f monit.log 
[UTC Apr 24 04:53:20] info     : 'consul_agent' trying to restart
[UTC Apr 24 04:53:20] info     : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl_wrapper
[UTC Apr 24 04:54:20] error    : 'consul_agent' failed to start
[UTC Apr 24 04:54:30] error    : 'consul_agent' process is not running
[UTC Apr 24 04:54:30] info     : 'consul_agent' trying to restart
[UTC Apr 24 04:54:30] info     : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl_wrapper
[UTC Apr 24 04:55:30] error    : 'consul_agent' failed to start
[UTC Apr 24 04:55:40] error    : 'consul_agent' process is not running
[UTC Apr 24 04:55:40] info     : 'consul_agent' trying to restart
[UTC Apr 24 04:55:40] info     : 'consul_agent' start: /var/vcap/jobs/consul_agent/bin/agent_ctl_wrapper

And:

vcap@agent-id-pcfdev-0:/var/vcap/sys/log/consul_agent$ tail -f consul_agent.stderr.log 
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"
error during start: timeout exceeded: "Unexpected response code: 500 (rpc error: failed to get conn: x509: certificate has expired or is not yet valid)"

PCF Dev version 0.29.0 (CLI: 49405ec, OVA: 0.548.0)

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[gmanoj79]

To temporarily fix the issue, I ssh'ed into cf dev (cf dev ssh) and changed the VM date/time to Apr 19th (sudo date -s "2018-04-19 00:00:00"). Seems like a certificate expired over the weekend.

[torstenwalter]

@gmanoj79 Thanks for sharing! I had the same issue. Setting the date back fixed it temporarily.

[ssisil]

Please follow #324 for the latest updates on this issue