rewrite-timestamp still rewrites base image layers (for some base images)

[AkihiroSuda]

rewrite-timestamp seems to still rewrites base image layers.

• Avoid applying SOURCE_DATE_EPOCH to base images #4663 was apparently not enough.

• This issue is similar to setting SOURCE_DATE_EPOCH changes the created metadata for base image layers (which are otherwise immutable 😅) #4614 , but this new issue is about the layers, not the metadata

Reproducer

Tested with BuildKit v0.13.1.

Dockerfile:

FROM busybox:1.36.1@sha256:650fd573e056b679a5110a70aabeb01e26b76e545ec4b9c70a9523f2dfaf18c6
RUN echo hi >/hi

$ buildctl  build \
  --frontend dockerfile.v0 \
  --local dockerfile=. --local context=. \
  --opt build-arg:SOURCE_DATE_EPOCH=0 \
  --opt platform=linux/amd64 \
  --output type=oci,dest=out,tar=false,rewrite-timestamp=true
[+] Building 3.3s (6/6) FINISHED                                                                                              
 => [internal] load build definition from Dockerfile                                                                     0.0s
 => => transferring dockerfile: 146B                                                                                     0.0s
 => [internal] load metadata for docker.io/library/busybox:1.36.1@sha256:650fd573e056b679a5110a70aabeb01e26b76e545ec4b9  2.1s
 => [internal] load .dockerignore                                                                                        0.0s
 => => transferring context: 2B                                                                                          0.0s
 => [1/2] FROM docker.io/library/busybox:1.36.1@sha256:650fd573e056b679a5110a70aabeb01e26b76e545ec4b9c70a9523f2dfaf18c6  0.8s
 => => resolve docker.io/library/busybox:1.36.1@sha256:650fd573e056b679a5110a70aabeb01e26b76e545ec4b9c70a9523f2dfaf18c6  0.0s
 => => sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c 2.15MB / 2.15MB                           0.8s
 => => extracting sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c                                0.1s
 => [2/2] RUN echo hi >/hi                                                                                               0.1s
 => exporting to oci image format                                                                                        0.2s
 => => exporting layers                                                                                                  0.0s
 => => rewriting layers with source-date-epoch 0 (1970-01-01 00:00:00 +0000 UTC)                                         0.2s
 => => exporting manifest sha256:bb6650298c854e7ce1e01184672c90ecd3bbd01279fc5de6693b3b19a2a4a20e                        0.0s
 => => exporting config sha256:f9fe9346991338c71f55c2e94b3003ab4741e04769e3cbedb998ed8da823f01a                          0.0s

manifest sha256:bb6650298c854e7ce1e01184672c90ecd3bbd01279fc5de6693b3b19a2a4a20e:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:f9fe9346991338c71f55c2e94b3003ab4741e04769e3cbedb998ed8da823f01a",
    "size": 583
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:e953e92b8f1e1ea41044293ccafab56864ecbe1de00864e96fc8475792786d16",
      "size": 2220178,
      "annotations": {
        "buildkit/rewritten-timestamp": "0"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:fe66163b0c83eaa75aa9f049875bff851a9a8a2e1ac03bd511e28aa91edf96c2",
      "size": 142,
      "annotations": {
        "buildkit/rewritten-timestamp": "0"
      }
    }
  ]
}

config sha256:f9fe9346991338c71f55c2e94b3003ab4741e04769e3cbedb998ed8da823f01a :

{
  "architecture": "amd64",
  "config": {
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Cmd": [
      "sh"
    ]
  },
  "created": "1970-01-01T00:00:00Z",
  "history": [
    {
      "created": "2023-05-18T22:34:17Z",
      "created_by": "BusyBox 1.36.1 (glibc), Debian 12"
    },
    {
      "created": "1970-01-01T00:00:00Z",
      "created_by": "RUN /bin/sh -c echo hi >/hi # buildkit",
      "comment": "buildkit.dockerfile.v0"
    }
  ],
  "os": "linux",
  "rootfs": {
    "type": "layers",
    "diff_ids": [
      "sha256:34bd552fa4b241684282886a3b112434867957c335a1528f5bd1c7f659a1e7ff",
      "sha256:1fe2590c45bef87a581fee096d92b04475131e5939e35f4c3b9b097c40df38d7"
    ]
  }
}

The output must retain the original layer sha256:7b2699543f22d5b8dc8d66a5873eb246767bca37232dee1e7a3b8c9956bceb0c (diff id sha256:95c4a60383f7b6eb6f7b8e153a07cd6e896de0476763bef39d0f6cf3400624b)

https://explore.ggcr.dev/?image=busybox%3A1.36.1%40sha256%3A650fd573e056b679a5110a70aabeb01e26b76e545ec4b9c70a9523f2dfaf18c6

[AkihiroSuda]

The issue does not seem to happen with FROM amd64/debian:bullseye-20230109-slim, so we couldn't catch this in the CI.