skaffold verify fails immediately without any useful logs

[nathanperkins]

Expected behavior

When using skaffold verify with the kubernetesCluster and the pod fails immediately, skaffold should give useful logs in the CLI or the Job and Pod should persist on the cluster so that they can be inspected.

I'd prefer to see the logs in the CLI, but if that is infeasible, it would be nice to have an option to keep skaffold from deleting the job and pod.

Actual behavior

Immediate skaffold verify failure results in no useful logs and there is no job or pod in the cluster. No logs are found in the GCP cloud logging console.

$ skaffold verify -a artifacts.txt
Tags used in verification:
 - <redacted> -> <redacted>:6489f2d-dirty
1 error(s) occurred:
* verify test failed: "<redacted>" running job "<redacted>" errored during run

Information

• Skaffold version: v2.13.0
• Operating system: Linux
• Installed via: skaffold.dev
• Contents of skaffold.yaml: (unfortunately proprietary)
• K8s cluster: GKE

[idsulik]

@nathanperkins hi! added some details within this PR #9589 , it should show fail reason and message now

[nathanperkins]

@idsulik that's awesome, thanks for following up with a quick improvement, it will definitely help.

I'm not sure if the pod.status.message is going to be able to show why the pod is crashing in all cases, though. I'm able to briefly see in the GKE console that the job exited with code 128 before it's deleted. I'm pretty sure the error will be in the logs. I could be wrong though.

I might be able to catch it if I'm quick enough, but what would really help the most is if I was able to prevent the job and pod from being deleted so I can inspect them freely.

[idsulik]

@nathanperkins , maybe you need this https://kubernetes.io/docs/concepts/workloads/controllers/ttlafterfinished/#cleanup-for-finished-jobs to keep the pod?
if you want to save logs into pod's data, then try this one terminationMessagePolicy https://kubernetes.io/docs/tasks/debug/debug-application/determine-reason-pod-failure/