skaffold dev fails to push image to minikube with non-default profile

[gintautassulskus]

Expected behavior

skaffold dev fails to deploy docker artifacts to minikube with a non-default profile.
skaffold deploy seems to be working as expected.

Information

• Skaffold version: v0.16.0
• Operating system: macOS 10.14
• Contents of skaffold.yaml: getting-started example

Steps to reproduce the behavior

FAILING:

1. minikube start --profile custom
2. cd skaffold/examples/getting-started
3. skaffold dev

Starting deploy...
kubectl client version: 1.12
pod/getting-started created
Deploy complete in 371.849422ms
ip-192-168-1-79:getting-started ginsul$ skaffold dev
Starting build...
Building [gcr.io/k8s-skaffold/skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM golang:1.10.1-alpine3.7 as builder
 ---> 52d894fca6d4
Step 2/6 : COPY main.go .
 ---> ccb384df2286
Step 3/6 : RUN go build -o /app main.go
 ---> Running in 2cba6e1bb0e4
 ---> 4b51cbb00758
Step 4/6 : FROM alpine:3.7
 ---> 34ea7509dcad
Step 5/6 : CMD ["./app"]
 ---> Using cache
 ---> 278b2babfb62
Step 6/6 : COPY --from=builder /app .
 ---> e74ca15d9d96
Successfully built e74ca15d9d96
Successfully tagged cd5f9db27bdd9047cfcff9e7151bff3c:latest
The push refers to repository [gcr.io/k8s-skaffold/skaffold-example]
d9063b240832: Preparing
ebf12965380b: Preparing
Cleaning up...
pod "getting-started" deleted
Cleanup complete in 4.429182568s
FATA[0011] exiting dev mode because the first build failed: building [gcr.io/k8s-skaffold/skaffold-example]: tagging: pushing: denied: Token exchange failed for project 'k8s-skaffold'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control 

eval $(minikube docker-env --profile custom) did not help.

[dgageot]

Hi @gintautassulskus Is this something you still experience with Skaffold v0.20.0?

[gintautassulskus]

Hi @dgageot, the issue persists with Skaffold v0.20.0. As a matter of fact, skaffold deploy no longer works producing the same error as skaffold dev.

[corneliusweig]

Hi @gintautassulskus , with the current master you can configure skaffold config set --global local-cluster true, so that image pushing will always be skipped. Does that solve your problem?

See #1355

[gintautassulskus]

Hi @corneliusweig, at the moment I am unable to test the use case as we are reworking our deployment system.
As long as one can deploy a demo app to the minikube with a custom profile this issue can be resolved.

[corneliusweig]

@gintautassulskus can you give this another try? If the issue is solved, you could close this.

[gintautassulskus]

@corneliusweig, the issue persists.

It seems that skaffold dev deploys only to default minikube profile. Even if I change the minikube profile via kubectl to another one, e.g. custom, skaffold still uses minukube.
I am not aware of any skaffold parameter to set target minikube profile.

[corneliusweig]

@gintautassulskus Have you made sure, that you are using the docker daemon from minikube and set the context as local? Iow

source <(minikube docker-env)
skaffold config set --global local-cluster true

If this is still not working, can you please provide further information from the following commands:

skaffold version
env | grep -i docker
kubectl config current-context
cat ~/.skaffold/config

[gintautassulskus]

The following sequence of commands:

minikube start --profile custom
source <(minikube docker-env --profile custom)
skaffold config set --global local-cluster true
cd skaffold/examples/getting-started
skaffold dev

Yields the following infinitely repeating pair of messages

rpc error: code = Unknown desc = Error response from daemon: manifest for gcr.io/k8s-skaffold/skaffold-example:7828be0b28ac8578e3e0cc99fcfc4196739d470794ecd5c90f31fb91a3bdf358 not found
Back-off pulling image "gcr.io/k8s-skaffold/skaffold-example:7828be0b28ac8578e3e0cc99fcfc4196739d470794ecd5c90f31fb91a3bdf358"

Does it work on your machine when you execute the given set of commands, @corneliusweig?

[corneliusweig]

@gintautassulskus I can't reproduce your error. The steps I took:

1. Start minikube with minikube start -p custom
2. Set up docker on host with source <(minikube docker-env -p custom --shell zsh)
3. cd skaffold/examples/getting-started
4. skaffold dev

More information:

• skaffold version -> v0.25.0
• env | grep docker -i

  DOCKER_TLS_VERIFY=1
  DOCKER_HOST=tcp://192.168.99.102:2376
  DOCKER_CERT_PATH=~/.minikube/certs
  DOCKER_API_VERSION=1.35
  

• kubectl config current-context -> custom
• minikube version -> 1.0.0
• cat ~/.skaffold/config

  global:
    local-cluster: true
  kubeContexts: []
  

Be sure to run all these setup commands in the same shell. If this does not work for you, please provide similar information about your environment as above.

[gintautassulskus]

@corneliusweig, finally, it worked. I had to use eval $(minikube docker-env -p custom) instead of source <(minikube docker-env -p custom --shell zsh), otherwise env | grep docker -i returned nothing.

The only diff from yours:

skaffold version
v0.26.0

[corneliusweig]

@gintautassulskus Maybe you want to chime in on #1957