-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VPC: allow configuring NAT instances instead of gateways (and a 0 NAT gateways bug) #1305
Comments
Thanks for reporting the issue about 0 NAT gateways, that's definitely something we should be looking into. My initial thought when you said you didn't want any NAT gateways was, you can make the But then you mentioned you DO want egress from a private subnet, but without a NAT gateway. I'm not an expert on VPCs, but I'm not entirely sure how that would even work. Are you talking about one-way UDP or ICMP packets only? Because surely TCP needs bidirectional packet flows (if only for ACKs) and so you'd need a NAT gateway to map between publicly routable and private IP addresses...? |
My assumption was that it would allow connections originating from the VPC outward (including TCP), but not the other way around. Looking at the doco, that seems not to hold strictly true:
In particular, it's only for IPv6, and recommends a NAT gateway for IPv4. The concept of the NAT gateway is great, the reality of the price is kind of terrible for smaller scale deployments. The 'old way' of achieving this (at least for me) pre-NAT gateway was to setup a NAT instance:
So I think what would be really nice here, is to have an easy wrapper around creating a NAT instance, and being able to use/specify it as part of the VPC Network, along with associated doco updates to point out the costs and that for smaller deployments that don't require the full power of the NAT gateways, that it might not be the best choice. As a default (and one that will create multiple by default), it's a costly mistake to have to make. Further reading: Steps:
The following setup might not be perfectly refined.. but it seems to be a way to implement the NAT instance pattern: const vpc = new ec2.VpcNetwork(this, 'Tokenized-VPC', {
natGateways: 0,
// natGatewayPlacement: {subnetName: 'Public'},
subnetConfiguration: [
{
cidrMask: 26,
name: 'Public',
subnetType: ec2.SubnetType.Public,
},
{
name: 'Application',
subnetType: ec2.SubnetType.Private,
},
],
defaultInstanceTenancy: ec2.DefaultInstanceTenancy.Default,
});
// Ref: https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-ec2.html#@aws-cdk/aws-ec2.SecurityGroup
const natSecurityGroup = new ec2.SecurityGroup(this, 'NATSecurityGroup', {
vpc,
groupName: "NATSecurityGroup",
description: "NAT Instance Security Group",
allowAllOutbound: true,
});
// TODO: Is this ok?
natSecurityGroup.connections.allowFromAnyIPv4(new ec2.TcpAllPorts());
natSecurityGroup.tags.setTag("Name", natSecurityGroup.path);
if (props.enableSSH) {
// Allow ssh access
// Ref: https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-ec2.html#allowing-connections
natSecurityGroup.connections.allowFromAnyIPv4(new ec2.TcpPort(22));
}
// TODO: Add AMI's for other regions here?
// const natImage = new ec2.GenericLinuxImage({
// "ap-southeast-2": "ami-062c04ec46aecd204", // amzn-ami-vpc-nat-hvm-2018.03.0.20181116-x86_64-ebs
// });
// Ref: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-instance.html
// Ref: https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-ec2.html#@aws-cdk/aws-ec2.SecurityGroupProps
const natInstance = new ec2.cloudformation.InstanceResource(this, 'NATInstance', {
imageId: "ami-062c04ec46aecd204", // ap-southeast-2: amzn-ami-vpc-nat-hvm-2018.03.0.20181116-x86_64-ebs
instanceType: new ec2.InstanceTypePair(props.ec2NATInstanceClass, props.ec2NATInstanceSize).toString(),
subnetId: vpc.publicSubnets[0].subnetId,
securityGroupIds: [natSecurityGroup.securityGroupId],
sourceDestCheck: false, // Required for NAT
keyName: "foo-ssh",
});
natInstance.propertyOverrides.tags = [
{key: "Name", value: `${natInstance.path}`}
];
// Route private subnets through the NAT instance
vpc.privateSubnets.forEach(subnet => {
const defaultRoute = subnet.findChild('DefaultRoute') as ec2.cloudformation.RouteResource;
defaultRoute.propertyOverrides.instanceId = natInstance.instanceId;
}); To route other subnet 'layers' with NAT (eg. if you use isolated): vpc.isolatedSubnets.forEach(subnet => {
const defaultRoute = subnet.findChild('DefaultRoute') as ec2.cloudformation.RouteResource;
defaultRoute.propertyOverrides.instanceId = natInstance.instanceId;
}); Later on I have an autoscaling group, and I allow SSH from the NAT gateway: const appAsg = new autoscaling.AutoScalingGroup(this, 'AppAutoScalingGroup', {
vpc,
minSize: 1,
maxSize: 1,
desiredCapacity: 1,
instanceType: new ec2.InstanceTypePair(props.ec2InstanceClass, props.ec2InstanceSize),
machineImage: amazonLinux2,
keyName: "foo-ssh",
vpcPlacement: {
subnetName: 'Application'
},
updateType: autoscaling.UpdateType.ReplacingUpdate,
});
if (props.enableSSH) {
// Allow ssh access
// Ref: https://awslabs.github.io/aws-cdk/refs/_aws-cdk_aws-ec2.html#allowing-connections
appAsg.connections.allowFrom(natSecurityGroup, new ec2.TcpPort(22))
}
const asgLaunchConfig = appAsg.findChild('LaunchConfig') as autoscaling.cloudformation.LaunchConfigurationResource;
asgLaunchConfig.addDependency(natInstance);
asgLaunchConfig.addDependency(appAsg.role); It would also be nice to have better wrappers around launching single EC2 instances (eg. to setup a bastion host/NAT instance/etc without having to drop to the cloudformation level), that has a similar interface to other higher level wrappers, and auto generates associated security groups/etc. I guess another alternative for my usecase might be to just put it in public and control access through security groups/etc. |
@0xdevalias -- for the cost conscious finding this issue we should also mention cross AZ network fees. If you really want to be least cost and run a single NAT (instance or GW) you should really use only one AZ. Depending on your data usage and instance sizes the cost implications will shift as well. |
Hi @0xdevalias I tried following your example, but it doesn't work in the newest version of the cdk clients. I did some changes to make the typing work as follows: const vpc = new VpcNetwork(stack, "MyApp", {
natGateways: 0
});
const natSecurityGroup = new SecurityGroup(stack, "NATSecurityGroup", {
vpc,
groupName: "NATSecurityGroup",
description: "NAT Instance Security Group",
allowAllOutbound: true
});
natSecurityGroup.tags.setTag("Name", natSecurityGroup.groupName);
natSecurityGroup.connections.allowFromAnyIPv4(new TcpAllPorts());
const natInstance = new CfnInstance(stack, "NATInstance", {
imageId: "ami-d03288a3",
instanceType: new InstanceTypePair(
InstanceClass.T2,
InstanceSize.None
).toString(),
subnetId: vpc.publicSubnets[0].subnetId,
securityGroupIds: [natSecurityGroup.securityGroupId],
sourceDestCheck: false, // Required for NAT
keyName: "myapp-ssh"
});
natInstance.propertyOverrides.tags = [
{ key: "Name", value: `${natInstance.stackPath}` }
];
vpc.privateSubnets.forEach(subnet => {
const defaultRoute = subnet.node.findChild("DefaultRoute") as CfnRoute;
defaultRoute.propertyOverrides.instanceId = natInstance.instanceId;
}); This yields the following error:
Any suggestions to how should I remove the default NAT Gateways and instead use my own NAT instance here? The default configuration is too expensive for us. We are trying to deploy a ECS stack (available via an ALB) which doesn't have much use for so many NAT Gateways. Thanks a lot |
Hi @sallar - I'm curious if you actually have a requirement for a NAT GW? If you don't, then you can accomplish this using only public subnets and not use a NAT GW. For example, if I had a very small deployment and I was hyper cost conscious and willing to trade some security layers I would just use public subnets and the Internet Gateway while controlling instance access via security groups. I might use a property override to disable the map public IP on launch as well (depending on some other requirements). You will have given up some ability with NACLs, but I don't see many people that need NACLs versus just security group control levels. If you do have an RDS need, remember that you can use an Just another thought on how to reduce this cost and keep most of the common security features. Full disclosure I am not an AWS engineer. |
Based on @sallar's snippet - this stack is working for me on AWS CDK 0.33. It creates a VPC with a t2.nano NAT instance without SSH access. I grabbed the instance ID from this page and looked up my corresponding AWS deployment region for the 'HVM (NAT) EBS-Backed 64-bit' instance. import cdk = require("@aws-cdk/cdk");
import ec2 = require("@aws-cdk/aws-ec2");
export class VpcStack extends cdk.Stack {
public readonly vpc: ec2.Vpc;
constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
super(scope, id, props);
this.vpc = new ec2.Vpc(this, "VPC", { natGateways: 0 });
const natSecurityGroup = new ec2.SecurityGroup(this, "NATSecurityGroup", {
vpc: this.vpc,
groupName: "NATSecurityGroup",
description: "NAT Instance Security Group",
allowAllOutbound: true
});
natSecurityGroup.connections.allowFromAnyIPv4(new ec2.TcpAllPorts());
const natInstance = new ec2.CfnInstance(this, "NATInstance", {
imageId: "ami-00c1445796bc0a29f",
instanceType: new ec2.InstanceTypePair(
ec2.InstanceClass.T2,
ec2.InstanceSize.Nano
).toString(),
subnetId: this.vpc.publicSubnets[0].subnetId,
securityGroupIds: [natSecurityGroup.securityGroupId],
sourceDestCheck: false // Required for NAT
});
natInstance.addPropertyOverride("Name", natInstance.stackPath);
this.vpc.privateSubnets.forEach(subnet => {
const defaultRoute = subnet.node.findChild(
"DefaultRoute"
) as ec2.CfnRoute;
defaultRoute.addPropertyOverride("instanceId", natInstance.instanceId);
});
}
} |
It really seems to be something broken with the This is my setup: const vpc = new ec2.Vpc(this, 'VPC', {
maxAzs: 1,
subnetConfiguration: [{
cidrMask: 24,
name: 'PublicSubnet',
subnetType: ec2.SubnetType.PUBLIC
}]
});
const cluster = new ecs.Cluster(this, 'Cluster', { vpc });
// Create a scheduled Fargate task
const collectorTask = new ecs_patterns.ScheduledFargateTask(this, 'FargateTask', {
cluster,
image: ecs.ContainerImage.fromRegistry(imageUri),
schedule: events.Schedule.expression(scheduleExpression)
}); When the
|
@Pwntus I think this might actually be a gap in the It's lacking a |
I have this same problem. I'm trying to deploy a CDK app to spin up an ECS on a nightly job and my biggest cost is going to be the NAT Gateway that I don't need to be using. |
If it's helpful, my setup is slightly different although it accomplishes much of the same thing. I do:
|
I'm pretty much in the same boat as you and I figured out how to get rid of the nat gateways. The trick is to put a cloudformation condition on the from aws_cdk.aws_ec2 import Vpc, CfnRoute
from aws_cdk.core import CfnCondition, Fn
vpc = Vpc(self, 'vpc', nat_gateways=0)
exclude_condition = CfnCondition(
self, 'exclude-default-route-subnet', expression=Fn.condition_equals(True, False)
)
for subnet in vpc.private_subnets:
for child in subnet.node.children:
if type(child) == CfnRoute:
route: CfnRoute = child
route.cfn_options.condition = exclude_condition # key point here @rix0rrr I think the above is a pretty reasonable workaround for this issue. |
I was able to use @jeshan's trick in order to create a VPC without a NAT gateway, but am now unable to create my ECS in a public subnet. It currently runs in a private subnet with no NAT gateway, which means it can't talk to anything. I'm still working on this. For those using javascript, @jeshan's code works as:
|
I was able to get the task to run in a public subnet. The trick is that configuration exists where you have the task configured to run, so in my case, it is done from a CloudWatch Event Rule, so the CDK code is:
|
Here is the complete CDK app I use that shows @jeshan's workaround in action: https://github.com/duo-labs/cloudmapper/blob/ec78ffe055692bf672bb60520a78378cccb4982a/auditor/lib/cloudmapperauditor-stack.js#L41 |
Thanks, additionally I needed to give my Fargate task a public IP: const cluster = new ecs.Cluster(this, "Cluster", { vpc });
const fargateTask = new sfn.Task(this, "Run Fargate Task", {
task: new sfnTasks.RunEcsFargateTask({
assignPublicIp: true,
cluster,
containerOverrides: [
…
],
integrationPattern: sfn.ServiceIntegrationPattern.SYNC,
taskDefinition,
subnets: { subnetType: ec2.SubnetType.PUBLIC }
}),
resultPath: "$.result"
}); |
I am facing an issue when creating a new VPC is the private subnet is required. Applied @0xdabbad00's suggestion but it still doesn't work Here is the error I am getting
Here is the code to construct the Vpc
Does anyone know how to fix this issue? |
Don't know if release 1.15 may fix the issue, https://github.com/aws/aws-cdk/releases/tag/v1.15.0
|
Yeah, I am using latest version 1.15.0 |
Adding subnetConfiguration solve my issue now
|
This seems to be possible with a recent commit #4898 in version 1.16.0 thanks to rix0rrr. new Vpc(this, `vpc`, {
cidr: '10.40.0.0/16',
maxAzs: 2,
natGateways: 2,
natGatewayProvider: NatProvider.instance({
instanceType: InstanceType.of(InstanceClass.T3A, InstanceSize.NANO),
}),
gatewayEndpoints: {
s3: { service: GatewayVpcEndpointAwsService.S3 },
},
}); |
Looks like there are two issues being discussed in this ticket. |
This is definitely resolved.
This is also resolved, but you must not request any private subnets (by definition, you must have at least one NAT traversal mechanism for private subnets). You should be creating isolated subnets instead. Closing this issue. Please reopen if I missed something. |
Hi all. @rix0rrr, I believe the 0 NAT gateway issue is not fixed. If I do const vpc = new Vpc(this, 'Vpc', {
cidr: '10.0.0.0/16',
maxAzs: 1,
natGateways: 0,
}); I get the following CDK error:
And if I do const vpc = new Vpc(this, 'Vpc', {
cidr: '10.0.0.0/16',
maxAzs: 1,
natGateways: 0,
subnetConfiguration: [
{
cidrMask: 24,
name: 'Public',
subnetType: SubnetType.PUBLIC,
}
],
}); I get the following CDK error:
What am I missing ? |
I seem to be having the same problem. |
I wasn´t able to completely get rid of the private subnets, but this code synthesized to a template without NAT gateways: vpc = ec2.Vpc(self, "VPC",
ip_addresses=ec2.IpAddresses.cidr('10.0.0.0/16'),
nat_gateways=0,
subnet_configuration=[
ec2.SubnetConfiguration(
name='Public',
cidr_mask=24,
subnet_type=ec2.SubnetType.PUBLIC,
),
ec2.SubnetConfiguration(
name='Private',
cidr_mask=24,
subnet_type=ec2.SubnetType.PRIVATE_WITH_EGRESS,
),
],
) |
When using
ec2.VpcNetwork
the defaults are to create NAT gateways. I originally scoped this down to just creating a single NAT gateway for my public subnet, and a month later was slogged with a $90 AWS bill, with almost all of that cost attributed to the NAT gateway.So today I decided to try and rework it to remove the NAT gateway (since my app really doesn't need it anyway). Tried removing the key, but that uses the defaults (and makes more), so I tried setting the key to
0
. Example config:When I ran
cdk diff
, I got a number of errors backThis implies that there isn't good support currently for when NAT gateways is 0 (may need to improve checks around things there), and as best as I could tell skimming the docs, there isn't a great way to use
VpcNetwork
without a NAT gateway.Presumably I can use the override methods to 'reach in' and patch those keys manually, probably setting GatewayId/EgressOnlyInternetGatewayId it will probably work, but I was wondering if there is currently a 'better' solution than that when:
It may be that most people using CDK have requirements greater than mine and/or don't mind about the NAT gateway costs, but I feel like someone just playing around may be shockingly surprised at how much $$ the defaults end up costing them. Maybe some doco changes to call this out more explicitly? And/or an example that supports using methods other than the NAT gateway (eg. as mentioned above, or an example that shows how to set it up using the old way with a NAT instance so we can run it on a micro for tiny workloads without costing the world)
The text was updated successfully, but these errors were encountered: