helm deploy of job

[rkjakeer]

Expected behavior

skaffold dev for deploy type helm with kubernetes job resources should be possible.

Actual behavior

skaffold dev terminates with error "The Job is invalid... field is immutable" when the job has to be redeployed. The first deploy/install with same chart will be successful. This behaviour is same as in issue #891 but with helm3.

Information

• Skaffold version: 1.5.0
• Operating system: Ubuntu 18.04.3
• example skaffold.yaml

Steps to reproduce the behavior

1. clone the skaffold repository and go to examples/helm-deployment
   https://github.com/GoogleContainerTools/skaffold/examples/helm-deployment

2. copy & paste below content to a file skaffold-helm/templates/job.yaml

apiVersion: batch/v1
kind: Job
metadata:
    name: {{ .Release.Name }}
    annotations:
        rollme: {{ randAlphaNum 5 | quote }}
spec:
  template:
    spec:
        containers:
        - name: hello
          image: busybox
          args: 
          - /bin/sh
          - -c
          - date; echo {{ .Release.Name }} running in the Kubernetes cluster
        restartPolicy: OnFailure

2. skaffold dev. the first deploy will run without issues.
3. When skaffold waits watching for changes, save/update anyfile in the helm chart to make skaffold redeploy the resources.
4. "Error: UPGRADE FAILED: failed to replace object: Job.batch "skaffold-helm" is invalid" will appear.

[balopat]

I dug into this a bit @rkjakeer, thanks for opening the issue!
It seems that helm 3.0 changed their behavior and we have to skip --force (helm/helm#7173) ... Helm's --force flag is wired up in Skaffold using --force=true. So, I managed to reproduce the issue based on your description, using Helm v3 and skaffold dev --force=false actually works!

Workaround
skaffold dev --force=false

Next steps

• We should figure out how to handle this more gracefully on our end - the change in --force between Helm v2/v3 should be reflected
• we should follow proposal: handle immutable resources on update helm/helm#7082 and other conversations around this so that we can align with the right semantics on Skaffold's end

[balopat]

Huh, actually this was reported before - closing as duplicate of #3798