Support NVMe SSD root volumes

[phillycheeze]

What would you like to be added:

The AWS EKS AMIs should support the root volume of NVMe SSD instance types within AWS.
When going through the Launch Instance flow of an eks-ami in the AWS console, it requires you to attach a second EBS volume on instance types that have an NVMe SSD root volume.

Why is this needed:

For workloads that require high disk-throughput, these instance types provide higher bandwidth over EBS volumes.

[Jeffwan]

/cc @Jeffwan

[leakingtapan]

If you uses Nitro instances (c5, m5, etc) the root volume will be NVMe by default:

[ec2-user@ip-172-31-21-94 ~]$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs         3890372       0   3890372   0% /dev
tmpfs            3903832       0   3903832   0% /dev/shm
tmpfs            3903832     292   3903540   1% /run
tmpfs            3903832       0   3903832   0% /sys/fs/cgroup
/dev/nvme0n1p1  20959212 1467484  19491728   8% /
tmpfs             780768       0    780768   0% /run/user/1000

Why do you need a second EBS volume?

[phillycheeze]

I'm talking about the d family instances specifically. Like the m5d.large instance type. These types don't use an EBS volume as a root device, instead they use a hardware-based SSD that is directly attached to the instance.

To be more specific, we use EKS for our Gitlab server inside of AWS. Our gitlab CI/CD runners use a dedicated EKS cluster. Since we are spinning up and down nodes many times throughout the day to match load of our CI jobs, we need only a small amount of disk space but very high throughput (docker pull/push and npm install being ran in parallel for dozens of jobs uses a ton of IOPs). It's more cost efficient for us to use the instance types that provide hardware NVMe SSDs as a root volume, and it seems like it's an easy thing to implement given that the Amazon Linux 2 OS provides driver support for these SSDs by default and the EKS AMIs are based off of them.

Please let me know if I'm missing something. From my testing, the AMIs require an EBS volumes with at least 20GB no matter the instance type.

[phillycheeze]

With that said, I imagine that using these instances probably already works without any configuration changes. This may simply be an AMI requirement for an EBS volumes that isn't necessarily required. If someone knows how to get around the requirement that the AMI have an EBS volume attached, I can test it and maybe use it without a problem.

[phillycheeze]

I've attached a screenshot that hopefully helps describe what I said above. It forces me to add an EBS volume on the 1.13.11 eks ami on an m5d.large instance type.

[phillycheeze]

So now I'm realizing that this isn't specific to EKS at all, that amazon doesn't allow the NVMe SSD to be the root device? If this is true, I'm probably just really dumb and you can close this issue :)

[leakingtapan]

With m5d family, NVMe volume is attached to each instance as ephemeral volume (non EBS volume).

With M5d instances, local NVMe-based SSDs are physically connected to the host server and provide block-level storage that is coupled to the lifetime of the M5 instance

See https://aws.amazon.com/ec2/instance-types/

Yep, and for this case NVMe won't be root device.

[phillycheeze]

Thanks so much!

[donotpush]

@phillycheeze we are facing the same issue, we are trying to use NVMe as root device to improve the performance of our Gitlab runners. Our current bottleneck is the IOPs. Did you manage to find a solution?

[johan13]

I had the same idea to put the root fs on NVMe for my GitLab runner. I haven't found a way to do it, but instead I just put /var/lib/docker on the NVMe volume. That is where all the disk access is anyway.

That can be done by adding "amazonec2-userdata=/usr/share/gitlab-runner/mount-nvme.sh" to MachineOptions in config.toml and use this mount-nvme.sh:

#!/bin/sh
mkfs.ext4 /dev/nvme0n1
mkdir -p /var/lib/docker
mount /dev/nvme0n1 /var/lib/docker

I use a Debian Buster AMI, the paths may be different on other distros.

[jae-63]

Thanks for this important clue @johan13 . We found that the following code fragment (based upon your suggestion) works for us, when inserted into our userdata, e.g. for r5d AWS instances. As you can see, in particular we found that we must stop/start the docker daemon.

if ( lsblk | fgrep -q nvme1n1 ); then
   mkfs.ext4 /dev/nvme1n1
   systemctl stop docker
   mkdir -p /var/lib/docker
   mount /dev/nvme1n1 /var/lib/docker
   chmod 711 /var/lib/docker
   systemctl start docker
fi

[wojtek-at-mambu]

Thanks a lot for the hints. I have one more finding on top of this thread.
The actual build in the runners happens in the /builds directory. This directory is a volume stored in /var/lib/kubelet.
The root filesystem of a container is indeed stored in /var/lib/docker.

This means:

• docker pulls go to /var/lib/docker
• git clone and build commands happen on /var/lib/kubelet
• some caches like gradle's /root/.gradle may still use /var/lib/docker

So to make the most of the SSD storage we use this preBootstrap snippet:

    preBootstrapCommands:
      - mkdir -p /mnt/data /var/lib/kubelet /var/lib/docker
      - mkfs.xfs /dev/nvme1n1
      - mount /dev/nvme1n1 /mnt/data
      - chmod 0755 /mnt/data
      - mv /var/lib/kubelet /mnt/data/
      - mv /var/lib/docker /mnt/data/
      - ln -sf /mnt/data/kubelet /var/lib/kubelet
      - ln -sf /mnt/data/docker /var/lib/docker

Tested on kubernetes versions 1.18 and 1.19, aws instance type c5d.4xlarge and gitlab runner chart 0.28.

[lonfme]

Thanks for this important clue @johan13 @jae-63 @wojtek-at-mambu.

https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html#launch-template-user-data

Amazon EC2 user data in launch templates:

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"
if ( lsblk | fgrep -q nvme1n1 ); then
    mkdir -p /mnt/data /var/lib/kubelet /var/lib/docker
    mkfs.xfs /dev/nvme1n1
    mount /dev/nvme1n1 /mnt/data
    chmod 0755 /mnt/data
    mv /var/lib/kubelet /mnt/data/
    mv /var/lib/docker /mnt/data/
    ln -sf /mnt/data/kubelet /var/lib/kubelet
    ln -sf /mnt/data/docker /var/lib/docker
fi

--==MYBOUNDARY==--

aws/containers-roadmap#596 (comment)

[bcouetil]

Thank you all for your inputs.

If I understand correctly, now containerd is the default container runtime (since Kubernetes 1.24).

Has anyone managed to do the same thing with containerd ?

[cartermckinnon]

@bcouetil we just merged a bootstrap flag + helper script that will do this for containerd and kubelet /var/lib-s. It'll ship in the next AMI release, more info in #1171

[kbumsik]

Thank you all for your inputs.

If I understand correctly, now containerd is the default container runtime (since Kubernetes 1.24).

Has anyone managed to do the same thing with containerd ?

So I swapped docker for containerd and it works for me.

You may want to use it until the next AMI release as @cartermckinnon mentioned.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
echo "Running custom user data script"
if ( lsblk | fgrep -q nvme1n1 ); then
  mkdir -p /mnt/data /var/lib/kubelet /var/lib/containerd
  mkfs.xfs /dev/nvme1n1
  mount /dev/nvme1n1 /mnt/data
  chmod 0755 /mnt/data
  mv /var/lib/kubelet /mnt/data/
  mv /var/lib/containerd /mnt/data/
  ln -sf /mnt/data/kubelet /var/lib/kubelet
  ln -sf /mnt/data/containerd /var/lib/containerd
fi

--==MYBOUNDARY==--

[primeroz]

for future reference if anyone lands here , the EKS AL2 and AL2023 amis provide a script to automate this

https://github.com/awslabs/amazon-eks-ami/blob/master/files/bin/setup-local-disks

which can also be run from the bootstrap script

[Alexhha]

https://github.com/awslabs/amazon-eks-ami/blob/master/files/bin/setup-local-disks

gives 404. Probably it was already removed and we should use LocalStorageOptions instead

https://github.com/awslabs/amazon-eks-ami/blob/main/nodeadm/doc/api.md#localstorageoptions