ARM64 aws-ebs-csi-driver images have x86_64 binaries

[sipsma]

/kind bug

What happened?
The arm64 aws-ebs-csi-driver images worked a few days ago but now all fail to run on arm64 machines. When I pull the images down and inspect them, they appear to all actually by x86 images even though they are labeled arm64. For example:

root@ip-10-0-0-243:~# ctr image pull --platform linux/arm64/v8 k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0
k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0:                                resolved       |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:a8ed0a90b49c16301f3ca934b75db63b8d5c8d68c377a865bce15c9af4a12ff5: done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:a5e67acf34d2f09cafca63fd41b6f9898f216ee98dde8ddf235d8edef660baf5:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:4ba180b702c8b29f4d24cb24eeb91a26a5924e797a806bbca1b73d26b5fe8b1e:    done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:5c3bea1f2f52c6b3cc8be0463f49ba95a2490377d0f848670d1af2ed2d4c6882:    done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:592701b57e30c44d07074686878f910b68eaa2d4afc75d3c6d53bb58c8807cbc:   done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.3 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/arm64/v8 sha256:a8ed0a90b49c16301f3ca934b75db63b8d5c8d68c377a865bce15c9af4a12ff5...
done
root@ip-10-0-0-243:~# mkdir tmp
root@ip-10-0-0-243:~# ctr image mount --platform linux/arm64/v8 k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.0 tmp
sha256:5bd728051e5a37f591e0acd0c50a51c6871e6953267f5be1c9021f892aca14a0
tmp
root@ip-10-0-0-243:~# file tmp/bin/aws-ebs-csi-driver 
tmp/bin/aws-ebs-csi-driver: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, Go BuildID=vkjo8nBXgvNN8K9RGjgz/qEsEhwoBGlBqbFS0hG3t/REDBNNkQ3nfX4Tmp1I6c/P8UIQWGuL
gHzj00Ilr00, stripped

What you expected to happen?
For the arm64 images to have arm64 binaries

How to reproduce it (as minimally and precisely as possible)?
See commands above using ctr or just try to run the image on an arm64 machine.

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version):
• Driver version:

[wongma7]

/assign

[wongma7]

Trying to verify if it's fixed: how do I use ctr images mount?

$ ctr --version
ctr github.com/containerd/containerd v1.3.9
$ ctr image mount --platform linux/arm64/v8 gcr.io/k8s-staging-provider-aws/aws-ebs-csi-driver:latest
No help topic for 'mount'

[wongma7]

NVM, it's on 1.4 branch.

[wongma7]

@sipsma does docker.io/amazon/aws-ebs-csi-driver:release-0.8-amazonlinux work for you? I am still testing the fix, trying to ensure that the gcr build process (prow) and docker hub build process (github actions) stay identical

[sipsma]

@wongma7 yes docker.io/amazon/aws-ebs-csi-driver:release-0.8-amazonlinux works for me now on an arm64 instance (the gcr one still fails like before)

[shrivastavshubham34]

@sipsma
Are you deploying them using kustomization overlay?
Because the below doesn't work for me

kubectl apply -k aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/arm64/

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
- ../../../base
images:
 - name: docker.io/amazon/aws-ebs-csi-driver
   newTag: "release-0.8-amazonlinux"
 - name: quay.io/k8scsi/csi-provisioner
   newName: raspbernetes/csi-external-provisioner
   newTag: "1.6.0"
 - name: quay.io/k8scsi/csi-attacher
   newName: raspbernetes/csi-external-attacher
   newTag: "2.2.0"
 - name: quay.io/k8scsi/livenessprobe
   newName: k8s.gcr.io/sig-storage/livenessprobe
   newTag: "v2.1.0"
 - name: quay.io/k8scsi/csi-node-driver-registrar
   newName: raspbernetes/csi-node-driver-registrar
   newTag: "1.3.0"

Neither does this

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
 - ../../base
images:
 - name: docker.io/amazon/aws-ebs-csi-driver
   newTag: release-0.8-amazonlinux
 - name: quay.io/k8scsi/csi-provisioner
   newTag: v1.5.0
 - name: quay.io/k8scsi/csi-attacher
   newTag: v2.2.0
 - name: quay.io/k8scsi/livenessprobe
   newTag: v2.1.0
 - name: quay.io/k8scsi/csi-node-driver-registrar
   newTag: v1.3.0

[sipsma]

@shrivastavshubham34 No, in my previous comment where I said the v0.8 amazonlinux image on dockerhub works I just meant that I could pull the image on an arm64 machine and see/execute arm64 binaries successfully. The kustomization overlays still don't work for me right now.

To temporarily workaround I ended up pulling this repo locally and modified some of the helm files to just use the 0.7.1 arm64 images (which don't have the same issues w/ arm64), leaving everything else the same. Seems to technically work but obviously far from ideal.

[shrivastavshubham34]

Thanks, @sipsma.
Since I need those clusters ASAP with EBS driver, I'll probably spin up an x86 cluster.

[wongma7]

0.8.1 is released/about to be released. It should be promoted from gcr.io/k8s-staging-provider-aws/aws-ebs-csi-driver:v20210106-v0.8.1 to k8s.gcr.io/provider-aws/aws-ebs-csi-driver:v0.8.1 in a few hours.

I tested that the staging images are multiarch following the steps in #666, so I would expect the final images in k8s.gcr.io to be as well, but I'm not going to close this until I get confirmation and confirm myself... I am not sure how the promotion actually works and whether it will maintain multiarchness of the images

[ayberk]

This should be resolved with the new release.

/close

[k8s-ci-robot]

@ayberk: Closing this issue.

In response to this:

This should be resolved with the new release.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.