Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove intermediate images and containers from local builds by default #1400

Merged
merged 15 commits into from
Apr 1, 2019

Conversation

nkubala
Copy link
Contributor

@nkubala nkubala commented Dec 18, 2018

This code changes the default behavior of the Skaffold runner to remove all intermediate images and containers from the local Docker daemon by default. Users can opt out of this by passing the --no-prune flag.

The local builder will keep track of all the image IDs it has built over the course of its run, and when the run is ended by the user, use the API client to remove each image. It will also pass the ForceRemove option to the Docker build to remove the intermediate containers from the build, e.g. in multi-stage builds.

Fixes #733

cc #1143 and #61

@dgageot dgageot self-assigned this Dec 18, 2018
Copy link
Contributor

@r2d4 r2d4 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

cmd/skaffold/app/cmd/dev.go Show resolved Hide resolved
Copy link
Contributor

@balopat balopat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks great! I'll test it out as well in a sec, in the meantime:

  1. What do you think about flipping the flag to become by default --prune=true and --prune=false for turning it off?
  2. Sorry that this is manual (I'll fix this soon) -- but updating the docs should be easy: if you run go run cmd/skaffold/man/man.go and replace the contents from ### skaffold build in the docs/content/en/docs/references/cli/_index.md that would be great.

cmd/skaffold/app/cmd/cmd.go Show resolved Hide resolved
@balopat
Copy link
Contributor

balopat commented Dec 19, 2018

After I tried it out, it feels flaky while doing skaffold dev on examples/microservices. Sometimes it prunes all the images, sometimes it fails:

Pruning images...
WARN[0003] builder cleanup: pruning images: Error response from daemon: conflict: unable to delete 4dafb0489464 (cannot be forced) - image is being used by running container 63b9248f91cb 

Maybe we should retry/wait somehow until the container exits?

@codecov-io
Copy link

codecov-io commented Dec 20, 2018

Codecov Report

Merging #1400 into master will decrease coverage by 2.01%.
The diff coverage is 6.83%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1400      +/-   ##
==========================================
- Coverage   51.45%   49.44%   -2.02%     
==========================================
  Files         171      174       +3     
  Lines        7690     8036     +346     
==========================================
+ Hits         3957     3973      +16     
- Misses       3357     3686     +329     
- Partials      376      377       +1
Impacted Files Coverage Δ
pkg/skaffold/plugin/builders/bazel/builder.go 15% <ø> (ø) ⬆️
pkg/skaffold/plugin/builders/docker/builder.go 22.8% <0%> (-4.28%) ⬇️
pkg/skaffold/debug/debug.go 44.82% <0%> (ø) ⬆️
pkg/skaffold/build/cache.go 0% <0%> (ø)
pkg/skaffold/plugin/environments/gcb/types.go 0% <0%> (ø) ⬆️
pkg/skaffold/plugin/shared/server.go 0% <0%> (ø) ⬆️
pkg/skaffold/docker/image_util.go 0% <0%> (ø)
pkg/skaffold/build/kaniko/types.go 0% <0%> (ø) ⬆️
pkg/skaffold/docker/parse.go 79.71% <0%> (ø) ⬆️
pkg/skaffold/plugin/builders/bazel/local.go 17.09% <0%> (-0.45%) ⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 01d8d7e...f19d89e. Read the comment docs.

@nkubala
Copy link
Contributor Author

nkubala commented Dec 20, 2018

After I tried it out, it feels flaky while doing skaffold dev on examples/microservices. Sometimes it prunes all the images, sometimes it fails

I ran through the microservices example 15 times, and I haven't been able to reproduce that issue. I'll go ahead and add a retry loop around it and see if it fixes the issue for you.

What do you think about flipping the flag to become by default --prune=true and --prune=false for turning it off?

I originally had it like this, but I liked --no-prune instead of --prune=false. It's similar to the --skip-tests flag, as opposed to --run-tests=false. The double-negative is only in the code, to the user it's more clear. WDYT?

@balopat
Copy link
Contributor

balopat commented Dec 21, 2018

It's a minor thing - we can keep it --no-prune.
Yeah, I'm happy to try with retries when you push it and the docs changes.

@dgageot
Copy link
Contributor

dgageot commented Jan 21, 2019

@nkubala Could you please rebase?

@nkubala
Copy link
Contributor Author

nkubala commented Mar 6, 2019

@balopat @dgageot @priyawadhwa this is RFAL. I reworked it to account for the builder plugins and the artifact cache: when caching, we won't prune all images built by skaffold (so we don't blow away the cache), but we WILL prune on cache eviction.

Copy link
Contributor

@balopat balopat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some comments after another look.
I'll have to still test it and understand the cache combination a bit better.

cmd/skaffold/app/cmd/cmd.go Outdated Show resolved Hide resolved
pkg/skaffold/build/bazel/bazel.go Outdated Show resolved Hide resolved
pkg/skaffold/build/local/types.go Outdated Show resolved Hide resolved
pkg/skaffold/build/plugin/core.go Outdated Show resolved Hide resolved
pkg/skaffold/runner/runner.go Outdated Show resolved Hide resolved
@nkubala nkubala force-pushed the prune branch 2 times, most recently from 21e8a59 to b666c73 Compare March 13, 2019 19:46
@@ -40,6 +40,8 @@ type SkaffoldOptions struct {
CacheArtifacts bool
ExperimentalGUI bool
EnableRPC bool
NoPrune bool
Prune bool // this is a logical combination of NoPrune and CacheArtifacts
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this is very confusing - what does it mean that it's a logical combination?
why do we need to store calculations in a "theoretically" read-only struct for user defined opts?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we could implement an instance function on it?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

opts.Prune()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

by logical combination I mean that opts.Prune == !opts.NoPrune && !opts.CacheArtifacts:

NoPrune CacheArtifacts Prune
F (default) F (default) T (default)
F (default) T F
T F (default) F
T T F

it's not really storing a calculation, it's just storing a simplified version of it to be used later on. to me this makes more sense than doing opts.Prune(), WDYT?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got the simplification - that part is fine - but I think we should introduce the IsPrune() function similar to Labels() instead of storing it in the struct to keep the design consistent i.e. "fields of SkaffoldOptions are user defined flags".

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

@nkubala nkubala merged commit 113cc47 into GoogleContainerTools:master Apr 1, 2019
@nkubala nkubala deleted the prune branch April 1, 2019 23:02
@tejal29 tejal29 mentioned this pull request Apr 12, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants