Support blockDeviceMappings with instance store NVMe volume to provisioned nodes

[maciej-ciach]

Version

Karpenter: v0.16.0

Kubernetes: v1.20.0

Expected Behavior

I would like to use EC2 instances with instance store volumes instead of provisioning additional EBS devices

Actual Behavior

I tried to configure provisioner using example of blockDeviceMapping from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/block-device-mapping-concepts.html#instance-block-device-mapping

    amiFamily: Bottlerocket
    blockDeviceMappings:
      - deviceName: /dev/xvdb
        virtualName: ephemeral0

But I'm getting provisioner errors when scheduling pods.

Steps to Reproduce the Problem

Configure provisioner with blockDeviceMappings block above

Resource Specs and Logs

[FernandoMiguel]

@maciej-ciach have you seen https://github.com/aws/karpenter/blob/074ed02/examples/provisioner/multiple-ebs.yaml ?

[bwagner5]

This would be a good thing to add! Probably not very difficult either if anyone would like to take a stab at a PR for it

[FernandoMiguel]

@bwagner5 add to where?
the code i linked is already in this repo

[bwagner5]

virtualName is not implemented yet and would be required for non-ebs instance storage.

[TechDawg270]

i will try to prep a PR for this over the next 2-3 days

[saurabhpal97]

Tried to fix in #2554

[d-m]

Is it possible to modify instance store volumes via blockDeviceMappings now? I see the PR for this was closed but not merged.

[cep21]

Are there workarounds or examples of how I can use the on host SSD for my ephemeral storage? Even if I need to make specific Provisioners, an example would be helpful

[bwagner5]

There's an open issue for this that has a little more activity, going to close this one as a duplicate and will track work on this front in #2723.

There is on-going work in the eks optimized AL2 AMI to setup a RAID-0 out of instance storage disks and remount containerd and kubelet. Once that PR is merged into the EKS Optimized AMI, we can then set the bootstrap flag within Karpenter to enable the new functionality and adjust the node ephemeral-storage capacity to assume that we'll use a RAID-0 setup for instance types with NVMe instance storage. awslabs/amazon-eks-ami#1171