Using --reproducible loads entire image into memory

[nathan-c]

Actual behavior
When running with --reproducible=true the entire image is loaded into memory.

Expected behavior
Memory profile should remain stable regardless of size of image being built.

To Reproduce
Steps to reproduce the behavior:

1. Create following Dockerfile

   FROM ubuntu:latest
   RUN dd if=/dev/zero of=output.dat  bs=1M  count=1000
   

2. Execute kaniko with the following parameters

   --context=/some-empty-directory
   --dockerfile=/some-empty-directory/Dockerfile
   --destination=<any destination>
   --cache=true
   --reproducible=true
   

3. Watch as memory consumption climbs dramatically near the end of the build.

Additional Information

• Having done some investigation, this is down to go-containerregistry not kaniko code so maybe this issue belongs there? The offending code is: https://github.com/google/go-containerregistry/blob/master/pkg/v1/mutate/mutate.go#L300
• Image used: gcr.io/kaniko-project/executor@sha256:fe07c91e9342a097a7ee7bf90d0d8b49b285a725315758e92de03e9f5debbb5c

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[nathan-c]

I should probably mention that this means we cannot use --reproducible for our builds because the containers running the build are being killed by k8s on our dev cluster and increasing the node RAM capacity is not a cost effective solution

[macrotex]

We are having this issue as well. We need to use reproducible builds so that our daily builds do not get added to our container registry if there is no "real" change to the docker image.

[darkdragon-001]

@Phylu Will this also be closed in the 1.7.0 release as of #1722?

[Phylu]

@Phylu Will this also be closed in the 1.7.0 release as of #1722?

I am not sure. Did not try this flag with my changes. If you are lucky, it could be that way.

[bh-tt]

@zx96 do you plan to create a PR for the linked commit? It would save many people a lot of frustration if our image builds would stop crashing at the end of a build due to being killed.

[zx96]

@zx96 do you plan to create a PR for the linked commit? It would save many people a lot of frustration if our image builds would stop crashing at the end of a build due to being killed.

I certainly can. I was planning on merging it shortly after I made the branch but ran into another bug (that also broke my use case) and got frustrated. I'll rebase my branch tomorrow and see if the other issue is still around - can't recall what issue number it was.

[zx96]

After rebasing my branch onto the latest main, it looks like I'm able to build an image with a 16GB test file in 512MB of memory with --compressed-caching=false --zero-file-timestamps! 🥳

#2477 opened - thanks for reminding me about this @bh-tt .

[philippe-granet]

With kaniko-project/executor:v1.19.2-debug, building the same image:

• with --reproducible flag, build took 5m 36s and use 7690Mb
• without --reproducible flag, build took 1m 38s and use 350Mb

Activing profiling (https://github.com/GoogleContainerTools/kaniko#kaniko-builds---profiling) I see lot of traces inflate/deflate with --reproducible flag :
kaniko.zip

[peter-volkov]

Still causes problems

[bh-tt]

To make the code causing the problem a bit clearer:

• during the DoBuild method, once the build is finished and reproducible flag is set, kaniko calls mutate.Canonical
• this function does 2 things: stripping out configuration (such as build hostname, docker version) and setting all layer times to epoch (through the Time function in the same file)
• the Time function iterates over each layer sets times to 0 through the layerTime helper function
• this function is where the problem lies: it creates a bytes.Buffer, uncompresses the original layer, reads each tar entry, changes timestamps and writes it to this buffer. A new layer is then returned backed by an in-memory Buffer

There are several options:

• 1. @zx96's strips out this metadata during creation of the original tar, which is likely the most efficient but appears to have some issues with layer caching?
• 2. replacing the memory-backed buffer with a file would solve Using --reproducible loads entire image into memory #862 but likely not help with --reproducible flag massively increases build time #1960 and may also decrease performance
• 3. minimum update would be to presize the in-memory buffer to be the same size as the original layer, which would likely reduce build time a little bit due to a reduced number of buffer copies, but I'm not sure this size is available
• 4. as an extra we could wrap the writer around the buffer with a gzip writer, which may reduce the size of the buffer somewhat (as we zip before writing to the buffer) and may create a small performance improvement as the tarball.ReadFromOpener may run gzip twice now, once for the hash and once for the compressed layer

I'm having some trouble running kaniko in my dev setup so it is hard to see the gains of 2-4, but until metadata is stripped out during build I think implementing them could save a lot of memory (at the cost of disk usage) and result in a small performance gain.

[bh-tt]

I've found a 5th option: submitting a PR to go-containerregistry making the layerTime function a lazy transformation. Since the problem lies in a dependency, that may be the best way to achieve this result.