helm deploy failed: matching build results to chart values: no build present for <image>

[WakeupTsai]

I'm trying to run the helm chart example, but seems the skaffold doesn't install the helm chart, and delete it directly.

INFO[0000] Skaffold &{Version:v0.24.0 ConfigVersion:skaffold/v1beta6 GitVersion: GitCommit:6a829c4b29e3a102b0b14c4584cd174f780402e9 GitTreeState:clean BuildDate:2019-03-01T19:38:41Z GoVersion:go1.10.8 Compiler:gc Platform:darwin/amd64}
WARN[0000] config version (skaffold/v1beta4) out of date: upgrading to latest (skaffold/v1beta6)
DEBU[0000] Defaulting build type to local build
INFO[0000] no config entry found for kube-context KUBE-CONTEXT
INFO[0000] Using kubectl context: KUBE-CONTEXT
INFO[0000] no config entry found for kube-context KUBE-CONTEXT
DEBU[0000] Using builder: local
DEBU[0000] setting Docker user agent to skaffold-v0.24.0
INFO[0000] no config entry found for kube-context KUBE-CONTEXT
DEBU[0000] push value not present, defaulting to true because localCluster is false
Starting build...
Build complete in 780.733µs
Starting test...
Test complete in 13.796µs
Starting deploy...
DEBU[0000] Executing template &{envTemplate 0xc4202bc900 0xc4200396c0  } with environment map[MY_ENV]
DEBU[0000] Running command: [helm --kube-context KUBE-CONTEXT get skaffold-helm]
Error: release: "skaffold-helm" not found
Helm release skaffold-helm not installed. Installing...
DEBU[0002] Executing template &{envTemplate 0xc4202bc300 0xc4202d05c0  } with environment map[MY_ENV]
Cleaning up...
DEBU[0002] Executing template &{envTemplate 0xc42057a100 0xc420039980  } with environment map[MY_ENV]
DEBU[0002] Running command: [helm --kube-context KUBE-CONTEXT delete skaffold-helm --purge]
Error: release: "skaffold-helm" not found
DEBU[0003] deleting release skaffold-helm: exit status 1
Cleanup complete in 1.882025206s
FATA[0003] exiting dev mode because first run failed: deploy failed: deploying skaffold-helm: matching build results to chart values: no build present for gcr.io/k8s-skaffold/skaffold-helm

[tejal29]

@WakeupTsai , i think the meaningful error is last line,
skaffold dev command failed to deploy due to error below and then tried deleting it since skaffold dev does a clean up before exiting. Does that make sense?

exiting dev mode because first run failed: deploy failed: deploying skaffold-helm: matching build results to chart values: no build present for gcr.io/k8s-skaffold/skaffold-helm"

Can you provide your helm Charts config or point us to your code?

[WakeupTsai]

I run the example code scaffold/examples/helm-deployment. By the way, I can install the helm chart manually.

~ ❯ helm version                                    
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.12.2", GitCommit:"7d2b0c73d734f6586ed222a567c5d103fed435be", GitTreeState:"clean"}

~ ❯ kubectl version  
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-30T21:39:38Z", GoVersion:"go1.11.1", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.8", GitCommit:"c138b85178156011dc934c2c9f4837476876fb07", GitTreeState:"clean", BuildDate:"2018-05-21T18:53:18Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

~ ❯ skaffold version  
v0.24.0

~ ❯ helm install --name test skaffold-helm      
NAME:   test
LAST DEPLOYED: Fri Mar  8 09:40:10 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Service
NAME                TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
test-skaffold-helm  ClusterIP  100.71.197.212  <none>       80/TCP   <invalid>

==> v1beta1/Deployment
NAME  DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
test  1        1        1           0          <invalid>

==> v1/Pod(related)
NAME                   READY  STATUS             RESTARTS  AGE
test-6c54d5bb79-n488r  0/1    ContainerCreating  0         <invalid>

[WakeupTsai]

After a few tests, I found the problem is caused by values in skaffold.yaml. But I don't know why this happens.

~ ❯ cat skaffold.yaml                                                                                                                                 
apiVersion: skaffold/v1beta6
kind: Config
deploy:
  helm:
    releases:
    - name: skaffold-helm
      chartPath: skaffold-helm
      values:
        image: gcr.io/k8s-skaffold/skaffold-helm

~ ❯ skaffold dev                                                                                                                                      
Starting build...
Build complete in 488.865µs
Starting test...
Test complete in 27.014µs
Starting deploy...
Error: release: "skaffold-helm" not found
Helm release skaffold-helm not installed. Installing...
Cleaning up...
Error: release: "skaffold-helm" not found
Cleanup complete in 1.899933618s
FATA[0003] exiting dev mode because first run failed: deploy failed: deploying skaffold-helm: matching build results to chart values: no build present for gcr.io/k8s-skaffold/skaffold-helm

~ ❯ cat skaffold.yaml    # remove the values                                                                                                                                   
apiVersion: skaffold/v1beta6
kind: Config
deploy:
  helm:
    releases:
    - name: skaffold-helm
      chartPath: skaffold-helm

~ ❯ skaffold dev                                                                                                                                            
Starting build...
Build complete in 472.87µs
Starting test...
Test complete in 14.874µs
Starting deploy...
Error: release: "skaffold-helm" not found
Helm release skaffold-helm not installed. Installing...
No requirements found in skaffold-helm/charts.
NAME:   skaffold-helm
LAST DEPLOYED: Fri Mar  8 10:09:03 2019
NAMESPACE: monitoring
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                            READY  STATUS             RESTARTS  AGE
skaffold-helm-65c5966796-29ltd  0/1    ContainerCreating  0         <invalid>

==> v1/Service
NAME                         TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
skaffold-helm-skaffold-helm  ClusterIP  100.64.127.181  <none>       80/TCP   <invalid>

==> v1beta1/Deployment
NAME           DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
skaffold-helm  1        1        1           0          <invalid>


Deploy complete in 7.618403877s
Watching for changes every 1s...
^CCleaning up...
release "skaffold-helm" deleted
Cleanup complete in 2.03769256s

[tejal29]

I am seeing this same error now on my branch where i am testing helm.
Assigning this to me

[tejal29]

Should be fixed in #2105.
We use to enforce Build in deploy flow. Somehow if the build result did not have the tag for the artifact in images, we would end up seeing this error.
I will verify this again on my branch to make sure this is fixed!

[alejandrox1]

Came across the same issue, tested it on v0.28 - v.033
This issue came up whenever i specified a namespace value inside of deploy.helm.releases[].values.

[tejal29]

We decided not to use #2105. I am no longer working on this.

[balopat]

@alejandrox1 can you share your setup?

Also, we have merged some fixes since then, it would be worthwhile trying to reproduce this on master / bleeding edge (https://skaffold.dev/docs/getting-started/#installing-skaffold).

[dgageot]

I'm going to assume this was fixed with the most recent changes to the helm deployer. Please reopen this issue if that's not the case. Thanks a lot for your contribution!

[petr-buchyn]

Please reopen this issue, I'm experiencing this issue with the newest bleeding-edge release.

[null93]

@petr-buchyn I am experiencing this issue as well. Setting my values with the setValueTemplates property rather than the values property seemed to serve as a good temporary workaround for me.

[lsimmons2]

@balopat I'm also having trouble installing a helm chart with skaffold. With helm 3 on microk8s with the skaffold.yaml below I get the following error:

apiVersion: skaffold/v2alpha2
kind: Config
metadata:
  name: server
build:
  artifacts:
  - image: cards-server
deploy:
  helm:
    releases:
    - name: foo
      chartPath: server-chart
      values:
        image: cards-server
        # tried below as well
        # image: localhost:32000/cards-server

~/org/cards/server$ skaffold dev
Listing files to watch...
 - cards-server
Generating tags...
 - cards-server -> localhost:32000/cards-server:e631c88-dirty
Checking cache...
 - cards-server: Found Remotely
Tags used in deployment:
 - cards-server -> localhost:32000/cards-server:e631c88-dirty@sha256:71e6c11615dc852b7d3f303347afd93554dd42c038c668013c4b668294847403
Starting deploy...
Cleaning up...
Error: unknown flag: --purge
FATA[0000] exiting dev mode because first deploy failed: deploying foo: matching build results to chart values: no build present for localhost:32000/cards-server

Other details:

~/org/cards/server$ helm version
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}
~/org/cards/server$ skaffold version
v1.2.0

• ubuntu 18
• using default skaffold registry - having run skaffold config set default-repo localhost:32000
• helm install works in installing the chart and skaffold dev works fine with a kubectl deployment.

[mauriciopoppe]

As a workaround I sent the values through a file referenced from valuesFiles

deploy:
  helm:
    releases:
      - name: r
        chartPath: path/to/helm/chart
        namespace: skaffold
        valuesFiles:
          - path/to/helm/values.yaml

[nkubala]

I have a feeling this issue wasn't really resolved the first time around, so I'm reopening until I feel a little more confident about it.

would anyone here be willing to try and repro this again on the latest skaffold version and tell me if the issue is still there? we've made some updated to helm in the last few months but I suspect there are still issues with the way the values are being passed around.

[abh]

Not sure if it's the same, but my issue in #3948 seems similar.

[aeneasr]

With a fresh kubernetes environment and Skaffold never being executed before, this fails when running dev because of the "release not found" error. For some reason Skaffold wants to delete a non-existing release and fails because it does not exist, during initial cleanup phase.
Mentioned workarounds do not work for me - the do work but only if --force=false is also provided.

eddit:// I'm on skaffold 1.7.0 and helm v3.1.2

[aeneasr]

Sorry, for the comment flood, did some more research:

      -
        name: kratos
        chartPath: ory/kratos
        remote: true # required for remote charts
        skipBuildDependencies: true # required for remote charts
        namespace: project-ory
        version: v0.2.0
        valuesFiles:
          - stack/projects/dist/ory/values.kratos.yaml
        imageStrategy:
          helm: {}
        values:
          tag: c1204da74d12

fails with:

[... build phase ...]
Successfully tagged gcr.io/ory-artifacts/backoffice:8f91bd7-dirty
time="2020-04-21T12:18:52+02:00" level=info msg="Build complete in 3m42.33562952s"
time="2020-04-21T12:18:52+02:00" level=info msg="Test complete in 1.183µs"
Tags used in deployment:
 - gcr.io/ory-artifacts/console -> gcr.io/ory-artifacts/console:e5b87deb6ed0e67d1b643d384c1a6f4ac640f6990d7fddc2c0e6f4641e399cf5
 - gcr.io/ory-artifacts/backoffice -> gcr.io/ory-artifacts/backoffice:c3e1c2646e1623d544bbabf328d04be50443b14bc6355dc8e93fc9ac2bc577c2
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
time="2020-04-21T12:18:52+02:00" level=debug msg="Running command: [helm version]"
time="2020-04-21T12:18:52+02:00" level=info msg="Deploying with helm v3.1.2 ..."
time="2020-04-21T12:18:52+02:00" level=debug msg="Running command: [helm --kube-context minikube-ory get all --namespace project-ory kratos]"
Helm release kratos not installed. Installing...

[ENV VARS REDACTED]

time="2020-04-21T12:18:53+02:00" level=debug msg="Running command: [helm --kube-context minikube-ory delete kratos --namespace project-ory]"
Error: uninstall: Release not loaded: kratos: release: not found
time="2020-04-21T12:18:53+02:00" level=warning msg="deployer cleanup: deleting \"kratos\": exit status 1"
time="2020-04-21T12:18:53+02:00" level=fatal msg="exiting dev mode because first deploy failed: deploying \"kratos\": release args: matching build results to chart values: no build present for oryd/kratos"

Removing the values does actually resolve the issue, so my previous comment is incorrect. This config worked before but I made these changes:

• I upgraded Helm from v2 to v3
• I upgraded Skaffold to 1.7.0 (I don't remember what version it was before, but timeframe was around Feb/March 2019)
• I replaced the Kratos build artifact (generated by Skaffold) with a Docker Image from Docker Hub

In another issue setting --force=false was recommended when using Helm v3 but I was not able to find any difference in output for this particular issue. This is actually incorrect, --force=false is in fact required for this error to disappear!

What's interesting though is that I have another release which uses an image built in the build phase which also uses values but doesn't cause the issue. Maybe this issue only exists for images not built locally?

      -
        name: backoffice
        chartPath: helm/charts/backoffice
        namespace: kernel
        valuesFiles:
          - skaffold/backoffice/values.yaml
        setValueTemplates:
          backoffice.config.executor.image.repository: "{{.IMAGE_NAME2}}"
          backoffice.config.executor.image.tag: "{{.DIGEST_HEX2}}"
        values:
          image: gcr.io/ory-artifacts/backoffice
        imageStrategy:
          helm: {}

In fact, I believe that this problem appears when the image is not built with Skaffold but pulled from e.g. Docker Hub. To summarize:

If an image is not built by Skaffold and the values key is set in the Skaffold Deploy Helm Config, Skaffold will fail to start.

I think it is legit to tag this as helm/v3 as well.

edit:// --force=false is actually required to get it working, on top of the mentioned

edit2://

This workaround does not fix the cleanup stage bug. It only allows skaffold to start. If you have a failed release or try to clean up with delete, Skaffold still fails.

1. Remove all values from any images that are being fetched from remote and not being built by Skaffold itself.
2. Run skaffold dev --cache-artifacts=true -v debug --force=false (I think --cache-artifacts=true is irrelevant but it speeds up builds)

[aeneasr]

Ok, sorry for digging this up! After some serious bug hunting it appeared that a chart we use upgraded but had also introduced regression issues. This caused the release to fail, culminating in the logs above. Because -v debug was set, I did not properly figure out why the build failed (due to the verbosity) and assumed that the cleanup was broken - which it isn't (as clearly indicated by the WARN instead of FATAL message). I also believe that I was hitting #3948 on top of #3798 which lead to further difficulties finding the root cause of the bug.

Now that everything is cleaned up, apart from the two linked issues, everything appears to be working. In fact, I believe that #1748 (comment) may cause #3948 . I believe that I've run into issues using the values chart last year as well.

In conclusion, if you end up with this issue, it's probably because your deployment failed due to a serious error (e.g. helm template broken, using wrong image, ...). If you check your helm releases using helm list and don't see the release in there, that's probably the root cause. Also omit -v debug in your CLI flags so that it becomes easier to identify the failure.

Thank you for your patience!

[tstromberg]

@aeneasr - thank you for the detailed feedback! I'm going to close this in the mean-time, but I suspect your research/debugging will prove useful for others to find.

If you can think of any ways for us to improve this error message, please open a new issue with the enhancement idea. Thank you!