Various "cf dev" commands report failure even though services appear to be working correctly

[nathan-alden-inl]

Machine information:

• Windows 7 x64
• Corporate proxy

Version information:

cf dev version: PCF Dev version 0.27.0 (CLI: 01c0c2b, OVA: 0.529.0)
cf version: 6.29.0+ff886fa.2017-07-24
VirtualBox: 5.1.26

I'm running into a strange situation installing and starting PCF Dev. All below commands were executed in PowerShell as an administrator.

Here is the output from cf dev start -k:

PS C:\Windows\system32> cf dev start -k
Using existing image.
Starting VM...
Provisioning VM...
Waiting for services to start...
8 out of 58 running
8 out of 58 running
8 out of 58 running
8 out of 58 running
41 out of 58 running
56 out of 58 running
58 out of 58 running
←[31;1mFAILED←[0m
Error: failed to provision VM: Process exited with: 127. Reason was:  ().

The strange thing is that I can still access the https://login.local.pcfdev.io website and interact with PCF Dev from cf:

PS C:\Windows\system32> cf login -a https://api.local.pcfdev.io --skip-ssl-validation
API endpoint: https://api.local.pcfdev.io

Email> admin

Password>
Authenticating...
OK

Select an org (or press enter to skip):
1. pcfdev-org
2. system

However, several commands do not work, despite the services apparently running just fine:

PS C:\Windows\system32> cf dev trust
←[31;1mFAILED←[0m
Error: PCF Dev is in an invalid state. Please run 'cf dev destroy' or 'cf dev stop'.

PS C:\Windows\system32> cf dev status
PCF Dev is in an invalid state. Please run 'cf dev destroy' or 'cf dev stop'

I am not sure how to proceed. I've seen some other GitHub issues that indicate I could SSH into the VirtualBox VM, but I haven't been able to locate the credentials to use at the Linux console.

Also, a co-worker is running PCF Dev 0.26.0 and does not experience these issues. Could this be a regression in the latest version?

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[nathan-alden-inl]

I tried SSHing into the VM with cf dev ssh from PowerShell but the prompt hangs after this text:

PS C:\Windows\system32> cf dev ssh
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.2.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Thu Aug 24 22:16:24 2017 from 192.168.11.1

From cmd.exe, it looks like this:

C:\Windows\system32>cf dev ssh
Your VM is currently stopped. Start VM to SSH to PCF Dev.

[nathan-alden-inl]

Although I could access https://login.local.pcfdev.io, I could not access https://apps.local.pcfdev.io. The error message displayed is 502 Bad Gateway: Registered endpoint failed to handle the request.. I'm guessing this has something to do with cf dev start failing.

[sclevine]

Hi @nathan-alden-inl,

Are you able to cf dev destroy and re-create the VM with cf dev start -k? (You will lose any data you have on VM.)

If so, can you provide the full output of that process?

[nathan-alden-inl]

Same output as before:

PS C:\Windows\system32> cf dev destroy; cf dev start -k
PCF Dev VM has been destroyed.
Using existing image.
Allocating 4096 MB out of 32648 MB total system memory (27073 MB free).
Importing VM...
Starting VM...
Provisioning VM...
Waiting for services to start...
7 out of 58 running
7 out of 58 running
7 out of 58 running
7 out of 58 running
40 out of 58 running
58 out of 58 running
←[31;1mFAILED←[0m
Error: failed to provision VM: Process exited with: 127. Reason was:  ().

[nathan-alden-inl]

Is there anything else I can run to get more detailed logging information? How can I SSH into the instance, give that cf def ssh doesn't seem to work?

[sclevine]

The SSH key is in ~/.pcfdev/vms/key.pem, but it seems like there's something seriously wrong with the VM. Can you describe your hardware? Are you running PCF Dev nested in another VM?

Given that you're behind an HTTP proxy, your VM won't have internet access unless you follow these instructions. That shouldn't really cause the issue you're seeing though.

If you run cf dev debug, strip out anything sensitive in the resulting archive, and send it to me at [email protected], I can attempt to figure out what's going on.

[nathan-alden-inl]

• Windows 7 x64
• Intel Core i7-6700
• 32 GB RAM
• 500 GB SSD HD

I am not running PCF Dev nested in another VM. I am running it within VirtualBox on the bare hardware of my workstation.

I've confirmed that, for the PCF components that do work, it is traversing the corporate proxy correctly. In my particular setup, I have Fiddler installed and acting as a proxy proxy to automatically handle authentication challenges from the corporate proxy. I've seen traffic going in and out of PCF Dev within Fiddler.

I will re-run cf dev with the debug flag and send you the scrubbed information. Thanks for helping!

[sclevine]

Thanks for sending me the debug archive. This appears to be a bug in 0.27.0 where Apps Manager doesn't start if you don't have a working internet connection when you run cf dev start before the VM has been created. This also causes issues with the shell when you cf dev ssh. Other functionality should not be affected.

Can you confirm that you've followed the instructions in the docs to provide PCF Dev with your proxy details? Starting with a working internet connection should let you workaround this issue.

[nathan-alden-inl]

I followed the instructions here: https://docs.pivotal.io/pcf-dev/proxy.html. As you can see in the debug files, I have the following three system-level environment variables set:

HTTP_PROXY=http://host.pcfdev.io:8888
HTTPS_PROXY=http://host.pcfdev.io:8888
NO_PROXY=.local.pcfdev.io,local.pcfdev.io,<redacted>

I am running Fiddler and have bound it to all IP addresses on my local machine (0.0.0.0), including 192.168.11.1, which is what host.pcfdev.io resolves to. Applications that use Windows' proxy configuration (Chrome, etc.) are successfully passing requests through Fiddler.

I am pretty sure all this means I did configure the proxy correctly.

This script:

cf dev destroy

$env:HTTP_PROXY
$env:HTTPS_PROXY
$env:NO_PROXY

cf dev start

runs with the following output (truncated at the end):

PS C:\Windows\system32> cf dev destroy
PCF Dev VM has been destroyed.
http://host.pcfdev.io:8888
http://host.pcfdev.io:8888
.local.pcfdev.io,local.pcfdev.io,<redacted>
Using existing image.
Allocating 4096 MB out of 32648 MB total system memory (27593 MB free).
Importing VM...
...

After looking at Fiddler, I don't see any requests coming from PCF. However, I do see one strange request that looks like it may be from PCF or the CF CLI:

GET http://0.0.0.0:9200/ HTTP/1.1
Host: 0.0.0.0:9200
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

This of course fails because the proxy doesn't allow a connection to 0.0.0.0. Could this be the source of the problem?

Another question for you: Does the CF CLI look only at user-level environment variables for HTTP_PROXY, HTTPS_PROXY, and NO_PROXY, or does it also look at system-level environment variables? Perhaps the issue is I am using system-level environment variables only.

[nathan-alden-inl]

I tried setting both the user-level and system-level environment variables at the same time, but it didn't seem to affect the problem.

[sclevine]

We're working on a follow-up release that will address the error that I found in your logs.

Are you able to start 0.26.0 successfully? If so, are you able to cf ssh into app containers in 0.26.0 and reach the internet?

[nathan-alden-inl]

Yes, I uninstalled 0.27.0 and installed 0.26.0 and everything seems to be working perfectly. I even have internet access from the VM:

PS C:\Windows\system32> cf dev ssh
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.2.0-42-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
vcap@agent-id-pcfdev-0:~$ curl https://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="https://www.google.com/">here</A>.
</BODY></HTML>

[sclevine]

Fixed in v0.28.0. Thanks for the report!

[srujanpulathota]

Hello,

can anyone help me on this.

i am trying to start the PCF dev, but running into below issues in windows 10.

PS C:\Users\sindh\AppData\Local\Packages\Microsoft.MicrosoftEdge_8wekyb3d8bbwe\TempState\Downloads> cf dev start -f .\pc
fdev-v1.2.0-windows.tgz
�[31;1mFAILED�[0m
cf dev start: checking for admin privileges: failed to execute: powershell.exe -Command "(New-Object Security.Principal.WindowsPrincipal([Security.Principal.WindowsIdentity]::GetCurrent())).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)": exec: "powershell.exe": executable file not found in %!P(MISSING)ATH%!:(MISSING)

Thanks
Srujan