skaffold port-forwarding does not throw error when port forwarding is not successful.

[tejal29]

I spend 3 hours trying to debug why my ruby application is not working on localhost:8080

Being new to ruby and trying someone's else code using skaffold (written by us), i immediately blamed the code and went down this rabbit hole to debug ruby code.

The actual error was, i had configured port-forwarding incorrecty.
My deployment name was "github-dash", where as in my skaffold config, my port-forward config,

portForward:
  - resourceType: deployment
    resourceName: github-dashing
    port: 3030
    localPort: 8080

From info logs, I could see port forwarding successful message.

Successfully built 7960275faea8
Successfully tagged github-dash:2662512-dirty
Tags used in deployment:
 - github-dash -> github-dash:7960275faea8a16c74ac8b40ea5ec992564e4c01e26616789b51e74815c2c347
Starting deploy...
 - deployment.apps/github-dash created
Waiting for deployments to stabilize...
 - deployment/github-dash is ready.
Deployments stabilized in 4.226963433s
Port forwarding deployment/github-dashing in namespace default, remote port 8080 -> address 127.0.0.1 port 3030
Press Ctrl+C to exit
Watching for changes...
[github-dash] Doing `require 'backports'` is deprecated and will not load any backport in the next major release.
[github-dash] Require just the needed backports instead, or 'backports/latest'.

Hitting localhost:8080 did not do anything.

Finally, after adding a bunch of debug statements, i ran a watch on kubectl port-forward command and realized there is not port-forward.
After turning trace on, i realized, port-forwding was not successful. I remember, we keep re-trying port-forwarding when it fails assuming it failed due to port not being available. We hoped it would eventually recover.
However, in this case, port-forwarding could never recover.
We need to change the logic for re-trying on only specific error conditions.

Deployments stabilized in 2.705981532s
DEBU[0004] getting client config for kubeContext: ``    
DEBU[0004] getting client config for kubeContext: ``    
Port forwarding deployment/github-dashing in namespace default, remote port 3030 -> address 127.0.0.1 port 8080
DEBU[0004] Running command: [kubectl --context minikube port-forward --pod-running-timeout 1s --namespace default deployment/github-dashing 8080:3030] 
Press Ctrl+C to exit
DEBU[0004] Change detected <nil>                        
TRAC[0004] Checking base image library/ruby:2.3.0 for ONBUILD triggers. 
DEBU[0004] Found dependencies for dockerfile: [{Gemfile /dashboard true} {. /dashboard false}] 
INFO[0004] Streaming logs from pod: github-dash-78bf88f6d5-nh6j6 container: github-dash 
DEBU[0004] Running command: [kubectl --context minikube logs --since=4s -f github-dash-78bf88f6d5-nh6j6 -c github-dash --namespace default] 
Watching for changes...
INFO[0004] Streaming logs from pod: github-dash-5699c97fd5-bj4wj container: github-dash 
DEBU[0004] Running command: [kubectl --context minikube logs --since=4s -f github-dash-5699c97fd5-bj4wj -c github-dash --namespace default] 
DEBU[0004] port forwarding deployment-github-dashing-default-3030 got terminated: exit status 1, output: Error from server (NotFound): deployments.apps "github-dashing" not found 

[IsaacPD]

skaffold/pkg/skaffold/kubernetes/portforward/entry_manager.go

Lines 113 to 125 in 5444e54

	b.entryForwarder.Forward(ctx, entry)
	color.Green.Fprintln(
	b.output,
	fmt.Sprintf("Port forwarding %s/%s in namespace %s, remote port %d -> address %s port %d",
	entry.resource.Type,
	entry.resource.Name,
	entry.resource.Namespace,
	entry.resource.Port,
	entry.resource.Address,
	entry.localPort))
	portForwardEvent(entry)
	}

That message is always printed out unconditionally despite whatever happens in b.entryForwarder.Forward(ctx, entry)

skaffold/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go

Lines 42 to 45 in 27a2d25

	type EntryForwarder interface {
	Forward(parentCtx context.Context, pfe *portForwardEntry)
	Terminate(p *portForwardEntry)
	}

There is no error returned to indicate that something went wrong. Currently the only implementation for EntryForwarder is KubectlForwarder which delegates everything to a go routine and just returns

skaffold/pkg/skaffold/kubernetes/portforward/kubectl_forwarder.go

Lines 73 to 77 in 27a2d25

	func (k *KubectlForwarder) Forward(parentCtx context.Context, pfe *portForwardEntry) {
	go k.forward(parentCtx, pfe)
	}
	func (k *KubectlForwarder) forward(parentCtx context.Context, pfe *portForwardEntry) {

In order to determine when to actually print the proper message instead of an error for kubectl, we must know when it succeeds so we should watch the logs for messages that indicate success (i.e. "Forwarding from 127.0.0.1:8080 -> 8080") and notify a channel as such.