NodeGroup with AL2 ami ending up with AL2023 userdata

[jdavredbeard]

What happened?

On the NodeGroup/V2 component, amiType, amiId, and operatingSystem are all optional arguments, and there is logic in the component to resolve the correct operating system from the NodeGroup component. However, if you pass an amiId for an AL2 ami, and don't specify amiType or operatingSystem, the NodeGroup assigns a default operating system of AL2023, which ends up generating the wrong userdata for the instances, and the instances are not able to join the cluster, but there is no deployment error.
This was discovered when trying to upgrade the pulumi-self-hosted-installers eks cluster from eks 2.8.1 to 3.3.0 without changing the nodegroup resource arguments.
I would expect amiId passed without amiType or operatingSystem to be invalid input.

Example

import * as aws from "@pulumi/aws";
import * as eks from "@pulumi/eks";
import * as pulumi from "@pulumi/pulumi";
import * as awsx from "@pulumi/awsx"

const vpc = new awsx.ec2.Vpc(`vpc`,
    {
        cidrBlock: "172.16.0.0/16",
        numberOfAvailabilityZones: 3,
        subnetStrategy: awsx.ec2.SubnetAllocationStrategy.Auto,
        natGateways: {
            strategy: awsx.ec2.NatGatewayStrategy.Single,
        },
    }
);

const cluster = new eks.Cluster("cluster", {
    vpcId: vpc.vpcId,
    authenticationMode: "API",
    providerCredentialOpts: { profileName: process.env.AWS_PROFILE}, 
    nodeAssociatePublicIpAddress: false,
    publicSubnetIds: vpc.publicSubnetIds,
    privateSubnetIds: vpc.privateSubnetIds,
    skipDefaultNodeGroup: true,
    createOidcProvider: false,
});

export const kubeconfig = pulumi.secret(cluster.kubeconfig.apply(JSON.stringify));

const managedPolicyArns: string[] = [
    "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy",
    "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy",
    "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly",
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
];

// Creates a role and attches the EKS worker node IAM managed policies
export function createRole(name: string): aws.iam.Role {
    const role = new aws.iam.Role(name, {
        assumeRolePolicy: aws.iam.assumeRolePolicyForPrincipal({
            Service: "ec2.amazonaws.com",
        }),
    });

    let counter = 0;
    for (const policy of managedPolicyArns) {
        // Create RolePolicyAttachment without returning it.
        const rpa = new aws.iam.RolePolicyAttachment(`${name}-policy-${counter++}`,
            { policyArn: policy, role: role },
        );
    }

    return role;
}

const role = createRole("ng-instanceProfile-test");
const instanceProfile = new aws.iam.InstanceProfile("ng-instanceProfile-test", {role: role});

const ssmParam = pulumi.output(aws.ssm.getParameter({
    // https://docs.aws.amazon.com/eks/latest/userguide/retrieve-ami-id.html
    name: `/aws/service/eks/optimized-ami/1.30/amazon-linux-2/recommended`,
}))
const amiId = ssmParam.value.apply(s => <string>JSON.parse(s).image_id)

new eks.NodeGroup("nodegroup", {
    cluster: cluster,
    nodeAssociatePublicIpAddress: false,
    nodeSecurityGroupId: cluster.nodeSecurityGroupId,
    clusterIngressRuleId: cluster.clusterIngressRuleId,
    instanceProfile: instanceProfile,
    amiId: amiId,
}, {
    providers: { kubernetes: cluster.provider},
});

new eks.NodeGroupV2("nodegroupV2", {
    cluster: cluster,
    nodeAssociatePublicIpAddress: false,
    nodeSecurityGroupId: cluster.nodeSecurityGroupId,
    clusterIngressRuleId: cluster.clusterIngressRuleId,
    instanceProfile: instanceProfile,
    amiId: amiId,
}, {
    providers: { kubernetes: cluster.provider},
});

Output of pulumi about

CLI
Version 3.142.0
Go Version go1.23.3
Go Compiler gc

Plugins
KIND NAME VERSION
resource aws 6.63.0
resource awsx 2.19.0
resource docker 4.5.7
resource docker 3.6.1
resource eks 3.3.0
resource kubernetes 4.18.3
language nodejs unknown

Host
OS darwin
Version 14.4
Arch arm64

This project is written in nodejs: executable='/Users/jdavenport/.nvm/versions/node/v18.20.4/bin/node' version='v18.20.4'

Current Stack: organization/NodeGroup-amiId/dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:dev::NodeGroup-amiId::pulumi:pulumi:Stack::NodeGroup-amiId-dev
pulumi:providers:aws urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:aws::default_6_63_0
pulumi:providers:awsx urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:awsx::default_2_19_0
awsx:ec2:Vpc urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc::vpc
aws:iam/role:Role urn:pulumi:dev::NodeGroup-amiId::aws:iam/role:Role::ng-instanceProfile-test
pulumi:providers:aws urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:aws::default_6_59_1
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:dev::NodeGroup-amiId::aws:iam/rolePolicyAttachment:RolePolicyAttachment::ng-instanceProfile-test-policy-3
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:dev::NodeGroup-amiId::aws:iam/rolePolicyAttachment:RolePolicyAttachment::ng-instanceProfile-test-policy-1
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:dev::NodeGroup-amiId::aws:iam/rolePolicyAttachment:RolePolicyAttachment::ng-instanceProfile-test-policy-0
aws:ec2/vpc:Vpc urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc::vpc
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:dev::NodeGroup-amiId::aws:iam/rolePolicyAttachment:RolePolicyAttachment::ng-instanceProfile-test-policy-2
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-3
aws:ec2/internetGateway:InternetGateway urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/internetGateway:InternetGateway::vpc
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-2
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-1
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-1
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-public-3
aws:ec2/subnet:Subnet urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet::vpc-private-2
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-1
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-3
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-2
aws:ec2/eip:Eip urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/eip:Eip::vpc-1
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-1
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-public-3
aws:ec2/routeTable:RouteTable urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable::vpc-private-2
aws:ec2/natGateway:NatGateway urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/natGateway:NatGateway::vpc-1
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-1
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-2
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-2
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-3
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-1
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-1
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-private-2
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-public-3
aws:ec2/routeTableAssociation:RouteTableAssociation urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/routeTableAssociation:RouteTableAssociation::vpc-public-3
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-2
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-1
aws:ec2/route:Route urn:pulumi:dev::NodeGroup-amiId::awsx:ec2:Vpc$aws:ec2/vpc:Vpc$aws:ec2/subnet:Subnet$aws:ec2/routeTable:RouteTable$aws:ec2/route:Route::vpc-private-3
pulumi:providers:pulumi urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:pulumi::default
pulumi:providers:eks urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:eks::default_3_3_0
eks:index:Cluster urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster::cluster
aws:iam/instanceProfile:InstanceProfile urn:pulumi:dev::NodeGroup-amiId::aws:iam/instanceProfile:InstanceProfile::ng-instanceProfile-test
eks:index:ServiceRole urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$eks:index:ServiceRole::cluster-eksRole
pulumi:providers:aws urn:pulumi:dev::NodeGroup-amiId::pulumi:providers:aws::default_6_47_0
aws:ec2/securityGroup:SecurityGroup urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::cluster-eksClusterSecurityGroup
aws:iam/role:Role urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$eks:index:ServiceRole$aws:iam/role:Role::cluster-eksRole-role
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksClusterInternetEgressRule
aws:iam/rolePolicyAttachment:RolePolicyAttachment urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$eks:index:ServiceRole$aws:iam/rolePolicyAttachment:RolePolicyAttachment::cluster-eksRole-4b490823
aws:eks/cluster:Cluster urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:eks/cluster:Cluster::cluster-eksCluster
pulumi:providers:kubernetes urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$pulumi:providers:kubernetes::cluster-eks-k8s
aws:ec2/securityGroup:SecurityGroup urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroup:SecurityGroup::cluster-nodeSecurityGroup
eks:index:VpcCniAddon urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$eks:index:VpcCniAddon::cluster-vpc-cni
aws:eks/addon:Addon urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:eks/addon:Addon::cluster-kube-proxy
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksNodeIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksExtApiServerClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksNodeClusterIngressRule
aws:ec2/securityGroupRule:SecurityGroupRule urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$aws:ec2/securityGroupRule:SecurityGroupRule::cluster-eksNodeInternetEgressRule
aws:eks/addon:Addon urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$eks:index:VpcCniAddon$aws:eks/addon:Addon::cluster-vpc-cni
pulumi:providers:kubernetes urn:pulumi:dev::NodeGroup-amiId::eks:index:Cluster$pulumi:providers:kubernetes::cluster-provider
eks:index:NodeGroupV2 urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroupV2::nodegroupV2
eks:index:NodeGroup urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroup::nodegroup
aws:ec2/launchTemplate:LaunchTemplate urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroupV2$aws:ec2/launchTemplate:LaunchTemplate::nodegroupV2-launchTemplate
eks:index:RandomSuffix urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroup$eks:index:RandomSuffix::nodegroup-cfnStackName
aws:ec2/launchConfiguration:LaunchConfiguration urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroup$aws:ec2/launchConfiguration:LaunchConfiguration::nodegroup-nodeLaunchConfiguration
aws:cloudformation/stack:Stack urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroup$aws:cloudformation/stack:Stack::nodegroup-nodes
aws:autoscaling/group:Group urn:pulumi:dev::NodeGroup-amiId::eks:index:NodeGroupV2$aws:autoscaling/group:Group::nodegroupV2

Found no pending operations associated with dev

Backend
Name jdavenports-MacBook-Pro.local
URL
User jdavenport
Organizations
Token type personal

Dependencies:
NAME VERSION
@pulumi/aws 6.63.0
@pulumi/awsx 2.19.0
@pulumi/eks 3.3.0
@pulumi/pulumi 3.142.0
@types/node 18.19.67
typescript 5.7.2

Pulumi locates its logs in /var/folders/68/1bv_wkxd4fgf81z7l3r37yy40000gn/T/ by default

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[flostadler]

Thanks @jdavredbeard for raising this. Right now users are expected to configure the operatingSystem in case they're configuring a custom amiId that's not the default OS. As part of pulumi-eks v3 we've changed the default OS to AL2023 (see migration guide).

I could see us requiring OS to be set when amiId is set and when custom user data is being created.
For NodeGroup/NodeGroupV2 that's always, for ManagedNodeGroup only in specific cases (i.e. when custom user data is created). But this would be a breaking change. I'm gonna track this as an enhancement.

For now we could add a warning for this case.