Cannot Prune Image, Being Used by Running Container

[ToucanBran]

Expected behavior

Exiting out of skaffold dev with --no-prune=false --cache-artifacts=false options set should prune images

Actual behavior

Received a warning message:

Cleaning up...
 - deployment.apps "psat-deployment" deleted
Pruning images...
time="2020-01-31T08:10:05-08:00" level=warning msg="builder cleanup: pruning images: Error response from daemon: conflict: unable to delete 124b637fbfc0 (cannot be forced) - image is being used by running container 65234d99ad2d"

Information

• Skaffold version: 1.3
• Operating system: Windows 10
• Contents of skaffold.yaml:

apiVersion: skaffold/v1beta2
kind: Config
build:
  local:
    push: false
  artifacts: 
    - image: private-repo.com/psat
      context: .
      docker:
        dockerfile: Dockerfile.dev
      sync:
        '**/*.js': .
deploy:
  kubectl:
    manifests:
      - "c:/src/k8s/psat/psat-deployment.yaml"

Steps to reproduce the behavior

1. Run a k8s cluster
2. For a given project. run skaffold dev --no-prune=false --cache-artifacts=false
3. Press ctrl+c

[nagasitaramthigulla]

@ToucanBran
you need to stop the running container and remove it before pruning the image.

[ToucanBran]

So if I'm developing and I run skaffold dev, I have to run a separate command to kill the container and then I can press Ctrl+c to have skaffold clean up everything? Seems kind of weird...

[tejal29]

@ToucanBran --no-prune=false should not prune images. I am able to reproduce your issue on getting started example.

tejaldesai@@getting-started (master)$ skaffold dev --no-prune=false --cache-artifacts=false
WARN[0000] port 50052 for gRPC HTTP server already in use: using 50053 instead 
Listing files to watch...
 - skaffold-example
Generating tags...
 - skaffold-example -> skaffold-example:v1.3.1-9-gbd157cca5-dirty
Found [minikube] context, using local docker daemon.
Building [skaffold-example]...
Sending build context to Docker daemon  3.072kB
Step 1/6 : FROM golang:1.12.9-alpine3.10 as builder
 ---> e0d646523991
Step 2/6 : COPY main.go .
 ---> Using cache
 ---> 315031316586
Step 3/6 : RUN go build -o /app main.go
 ---> Using cache
 ---> b30771784f6d
Step 4/6 : FROM alpine:3.10
 ---> af341ccd2df8
Step 5/6 : CMD ["./app"]
 ---> Using cache
 ---> 3ae235f3b181
Step 6/6 : COPY --from=builder /app .
 ---> Using cache
 ---> 79a8d1925f97
Successfully built 79a8d1925f97
Successfully tagged skaffold-example:v1.3.1-9-gbd157cca5-dirty
Tags used in deployment:
 - skaffold-example -> skaffold-example:79a8d1925f97436dbda213ec5b553909f28c11a3c2e1f0fbbecdc76804a01f9b
   local images can't be referenced by digest. They are tagged and referenced by a unique ID instead
Starting deploy...
 - pod/getting-started created
Watching for changes...
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
[getting-started] Hello world!
^CCleaning up...
 - pod "getting-started" deleted
Pruning images...
WARN[0013] builder cleanup: pruning images: Error response from daemon: conflict: unable to delete 79a8d1925f97 (cannot be forced) - image is being used by running container fa02b2a91420 
tejaldesai@@getting-started (master)$ 

[tejal29]

@ToucanBran Sorry, i read the --no-prune flag wrong.
As per your command the images should be pruned. Looks like there is a race condition between the local docker daemon and it still thinks the pod is using the image.

[tejal29]

ok. i think i found how to reproduce this error on master:

1. Run generate-pipeline example

cd examples/generate-pipeline
$ skaffold run

2. Now, run the following

eval $(minikube docker-env)

3. Run

docker ps

4. You shd see following line in the output

CONTAINER ID        IMAGE                          COMMAND                  CREATED             STATUS              PORTS               NAMES
dc1feea173f7        e815e8a41d7e                   "./app"                  14 seconds ago      Up 14 seconds                           k8s_generate-pipeline_generate-pipeline_default_4d21d5d0-46d1-11ea-acb4-187613dde27e_0

4. Keep this app running.

5. Open another terminal and run

watch docker ps

6. Now go to examples/getting-started

cd ../examples/getting-started

7. Run skaffold dev and hit control C

skaffold dev --no-prune=false --cache-artifacts=false
....

8. You will see docker container for getting-started pod with the same image id e815e8a41d7e

docker ps
CONTAINER ID        IMAGE                          COMMAND                  CREATED              STATUS              PORTS               NAMES
6dbbf3da7b6f        e815e8a41d7e                   "./app"                  53 seconds ago       Up 52 seconds                           k8s_getting-started_getting-started_default_54d48025-46d2-11ea-acb4-187613dde27e_0
... 
k8s_POD_getting-started_default_54d48025-46d2-11ea-acb4-187613dde27e_0
ccc4bc4bfe9b        e815e8a41d7e                   "./app"                  About a minute ago   Up About a minute                       k8s_generate-pipeline_generate-pipeline_default_494f7970-46d2-11ea-acb4-187613dde27e_0

9. Hit Cntrl C on the skaffold dev

[getting-started] Hello world!
^CCleaning up...
 - pod "getting-started" deleted
Pruning images...
WARN[0127] Error response from daemon: conflict: unable to delete e815e8a41d7e (cannot be forced) - image is being used by running container ccc4bc4bfe9b 

Looks like, the image id calculated is same for both getting-started and generate-pipeline example.
@ToucanBran Do you have another skaffold run or skaffold dev process running?

[ToucanBran]

@tejal29 I do not, just the single skaffold dev running as far as I know. I'm not using minikube if that makes a difference. I'm on a windows machine using Docker Desktop

[tejal29]

Thanks @ToucanBran. Can you run docker ps | grep fa02b2a91420 ?

[ToucanBran]

do you want me to run your instructions above or run it with my app?