nerdctl image save fails when discard_unpacked_layers=true is enabled

[2rigor]

Description

When GC layers optimization is enabled (discard_unpacked_layers=true) some/all layers may be missing from containerd content storage.
Thus nerdctl image save fails to export image. Can CLI be fixed to export image content? Practically the content is present on the disk

Similarly to containerd/containerd#9277

Steps to reproduce the issue

1. Install K8s with containerd
2. Download and install nerdctl (from github)
3. Enable discard_unpacked_layers=true in /etc/containerd/config.toml
   Alternatively you may create e.g. GKE 1.27
4. list images using sudo nerdctl -n k8s.io images, choose some image. Instead run pod with some image e.g. alpine:latest
   sudo nerdctl -n k8s.io image save <image_ref> > /dev/null e.g. sudo nerdctl -n k8s.io image save docker.io/library/alpine:latest > /dev/null

Describe the results you received and expected

You will get error:
FATA[0000] failed to get reader: content digest sha256:89d9c30c1d48bac627e5c6cb0d1ed1eec28e7dbdfbcc04712e4c79c0f83faf17: not found

Expected - no failure (and to get tar file)

What version of nerdctl are you using?

Client:
Version: v1.6.2
OS/Arch: linux/amd64
Git commit: e3dc23b
buildctl:
Version:

Server:
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

Client:
Namespace: default
Debug Mode: false

Server:
Server Version: 1.6.24
Storage Driver: overlayfs
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Log: fluentd journald json-file syslog
Storage: native overlayfs
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-84-generic
Operating System: Ubuntu 22.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 4.798GiB
Name: rigor-ctrd-42
ID: abe6dcae-815f-4437-845d-09e1ef8851ee

[dineshp-09]

Any updates on this? The issue is reproducible on GKE version 1.27 and above and EKS version 1.26 and above.

[2rigor]

Any updates on this? The issue is reproducible on GKE version 1.27 and above and EKS version 1.26 and above.

In fact, @dineshp-09, based on the below it should probably hannep on any support EKS with recent AMIs (since Jul-28 this year): https://github.com/awslabs/amazon-eks-ami/blob/5b54e37cde1e8a4c67a3ddaff27c953fc7b7b37f/CHANGELOG.md?plain=1#L270