[Design] enable internal ALBs for Backend Services

[huanjani]

Add support for a shared internal load balancer with Backend Services

Allow users to add the http field, currently limited to internet-facing Load Balanced Web Services, to Backend Services' manifests. The deployment of a Backend Service with http specified will trigger the creation of an internal load balancer in the environment to which the service is deployed. The internal load balancer may be shared by all Backend Services in the environment.

name: api
type: Backend Service

http:
  path: '/'
...

For users who want to have an https endpoint, they can leverage the new import existing certificates feature. If certs are imported into an environment with a VPC with only private subnets, they will be automatically applied to the internal load balancer.

$ copilot env init --import-vpc-id vpc-099x12d2b34abcd47 \
--import-private-subnets subnet-099fafef48fb3c547,subnet-00c9e76f123456e7f \
--import-cert-arns arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012 

Users can then bring their own hosted zones so that once the load balancer is created, Copilot will insert the alias record into the hosted zone, mapping the alias name to the LB's DNS name.

http:
    path: '/'
    alias: example.com # will work only for environments that have imported certificates 
    hosted_zone: Z0873220N255IR3MTNR4 # alias requires a `hosted_zone`

When multiple services and aliases are in play, users will need to 1. bring a hosted zone and provide a unique alias for each service they create or 2. not bring a hosted zone and add records post-release, independently of Copilot.

Related: #2961, #3344.

[huanjani]

Q for those interested in this feature:
Would you want

1. all resources within the VPC to have access to the internal load balancer, or
2. only Copilot services to have access?
   Thanks!

[rverpillot]

We need internal ALB, but we will have to use a VPC with "public" subnets routed on our on-prem network and "private" subnets for the application only. So the ALB should be deployed on the "public" subnets to be reachable from the on-prem network, whereas ECS services will be deployed on private subnets.

So unfortunately, I think this design will be not compatible with our needs.

[efekarakus]

we will have to use a VPC with "public" subnets routed on our on-prem network and "private" subnets for the application only.

Hi @rverpillot ! I have a few questions below if you don't mind to help us build the right thing:

1. Oh interesting, so the VPC has subnets that have a route to an internet gateway?
   I'm trying to understand if Copilot will detect these subnets as public or if it will think that they're private. Today, a subnet is defined as public if and only if it has a route to an IGW.

2. If there is no IGW, then does it mean you'd like to control which subnets to place the ALB in?

3. If there is an IGW, it sounds like you want to still place the ALB in the public subnets while keeping it's scheme as internal is that correct? Is this due to how the on-prem network is connected with the subnets - what do you use to establish the connection?

Thank you!

[rverpillot]

Bonjour @efekarakus ! It’s fine to help you to help us !

All these subnets are private (no IGW). But importing them in copilot as public and private subnets permits to keep the network placement policy. In this case “public” means “enterprise scope” and “private” means “application scope”.

Maybe I’m wrong but Copilot doesn’t check an existing IGW by itself for public subnets, but AWS API does it just before to create an internet-facing ALB (obviously it’s not the case for an internal ALB).

[huanjani]

Hello, @rverpillot!

Just to clarify, please:

1. You would like to use your enterprise-scope subnets that don't have IGWs; can you import them to your Copilot env as what Copilot refers to as "private"? What do you mean by "network placement policy"?

2. If the internal ALB's security group has inbound rules for 80 and 443 ports and source from 0.0.0.0/0, does that satisfy your requirement for your on-prem network (enterprise scope) AND application scope?

Thank you so much. I apologize for the back and forth; we just want to make sure we are building what you need.

[rverpillot]

Hello @huanjani .

We have 2 private subnets for the application deployment: 1 for the services only (and not directly accessible from outside), and 1 where the ALB must be deployed to be accessible from our on-prem network (because it will get a routable IP address through AWS Transit Gateway and AWS Direct Connect).

So by importing our "enterprise-scope" subnets as "public" subnets and our "application-scope" subnets as "private" subnets in the Copilot environment, the internal ALB could be deployed in the right place, and we can still using the network.vpc.placement attribute to place the services in "private" subnets (with maybe some exceptions).

The Inbound rules for 80 and 443 ports and source from 0.0.0.0/0 are OK for me.

Maybe I could post a diagram to clarify this ?

[huanjani]

A diagram would be great! Thank you!

[rverpillot]

Maybe it's not clear because we are not talking about the same things. I understand this design is to add an internal ALB for backend services only. But what about frontend services ?

Features we need to deploy copilot applications on private VPC (multiple private subnets and no IGW):

• importing VPC and subnets.
• importing user certificates.
• creating internal ALB instead internet-facing ALB in the subnets called "public" (or add a new setting to choose the placement).
• adding DNS records into private hosted zones.

[rverpillot]

[huanjani]

Thank you so much for the additional information!

Our plan is what you mentioned above, to "add a new setting to choose the placement." This is because we'd like to keep Load Balanced Web Service to be reserved for internet-facing services and Backend Service to model any internal services.
For your use case, you will:

1. Run env init, using a (new) flag to indicate subnet placement for the internal ALB, and importing all of your subnets as "private" (for example: env init --internal-alb-subnets)
2. Deploy all of your services as "backend" and in their manifests specify

type: 'Backend Service'
http:
  path: '/'
  alias: example.com # will work only for environments that have imported certificates 
  hosted_zone: Z0873220N255IR3MTNR4 # alias requires a `hosted_zone`
vpc:
  placement:
    subnets: [either by subnet IDs or by tags] 

In this way, you can place the internal ALB in the enterprise scope, and your services in the application scope, but the services will have access to the internal load balancer.

Please let us know if you have further questions or input!

[rverpillot]

Just 1 question: do you confirm the only difference between "Backend Service" with http field and "Balanced Web Service" will be the scheme of the ALB serving them ? All HTTP features will be available for backend services (healthcheck, stickiness, ...) ?

[huanjani]

Yes!
We didn't include all the fields in the sample above, to keep things simple, but it will be like:

  http:
    path: '/'
    healthcheck:
      path: '/_healthcheck'
      success_codes: '200,301'
      healthy_threshold: 3
      unhealthy_threshold: 2
      interval: 15s
      timeout: 10s
      grace_period: 45s
    deregistration_delay: 5s
    stickiness: false
    allowed_source_ips: ["10.24.34.0/23"]
    alias: example.com //  will work only for environments that have imported certificates 
    hosted_zone: Z0873220N255IR3MTNR4

[rverpillot]

@huanjani we are OK with this design.

Maybe in the future, these 2 kinds of service could be merged, and only network placement and maybe an alb attribute could be use to setup an internal or an internet-facing ALB.

whatever, thank for your listening.

[rverpillot]

Bonjour @huanjani

The feature hosted_zone_id is not available yet, right ?

[huanjani]

Bonjour!
It is, but the field is now just hosted_zone; sorry about that-- I will change it above.

[rverpillot]

It works. Thank you @huanjani !

But I found an another issue: the security group of the internal ALB is not correct: inbound rules for port 80 and 443 are missing.

[huanjani]

Yes, we acknowledge this request, and the feedback that we got supported that request as well, but we decided to build the permissions as least-privileged, knowing that users may optionally add permissions using addons. This will make the feature more flexible and secure.

Your ./copilot/svc/addons/template.yml might look like:

Parameters:
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.
  Name:
    Type: String
    Description: The name of the service, job, or workflow being deployed.
Resources:
  InternalLoadBalancerSecurityGroupIngressFromHttp:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow anyone on port 80
      CidrIp: 0.0.0.0/0
      FromPort: 80
      ToPort: 80
      IpProtocol: tcp
      GroupId:
        Fn::ImportValue:
          !Sub '${App}-${Env}-InternalLoadBalancerSecurityGroup'
  InternalLoadBalancerSecurityGroupIngressFromHttps:
    Type: 'AWS::EC2::SecurityGroupIngress'
    Properties:
      Description: Allow anyone on port 443
      CidrIp: 0.0.0.0/0
      FromPort: 443
      ToPort: 443
      IpProtocol: tcp
      GroupId:
        Fn::ImportValue:
          !Sub '${App}-${Env}-InternalLoadBalancerSecurityGroup'

Also, in the next few weeks, we will introduce environment manifests 🎉 , which will allow you to configure the security group of the ALB, giving you an easier way to set permissions!

[adrienlauer]

Hi!

Regarding the security group permissions, would it be possible to transparently open 80 and 443 only if an http section is defined in the manifest ?

I understand the need to follow the least-privileged principle but if you declare an http section, you definitely expect HTTP(S) to be allowed. This would keep behavior differences between Load-Balanced Web Service and Backend Service types at a minimum, which is a good thing for us.

What do you think about it, @huanjani ?

[huanjani]

Hello, @adrienlauer!

We totally understand the desire to streamline the opening up of permissions. We will now add a flag to be used with copilot env init that will allow traffic to the internal ALB from ports 80 and 443.

Hope that works for you!

[rverpillot]

Hello @huanjani

I understand you don't want to add these inbound rules by default to the security group when the internal ALB is used between Copilot services only because it would be a security issue. But could be possible to detect automatically the use case where the internal ALB is used like a frontend ALB and allow traffic from ports 80 and 443 ?

[efekarakus]

Bonsoir @rverpillot !

We have merged fbd454d which adds the flag copilot env init --internal-alb-allow-vpc-ingress that allows ingress from port 80 and 443. Is that okay with you?

This is a short-term workaround. In July, we're planning to deliver #3522 which will allow you to modify environment resources such as load balancer security groups, add or remove certificates, ECS cluster settings. The env manifest should make using these flags obsolete

[rverpillot]

Bonjour @efekarakus !

Yes, it’s OK for us. I already built copilot with this commit and we will test this feature the next week.

Thanks.

[huanjani]

This feature is now released in v1.19.0: https://github.com/aws/copilot-cli/releases/tag/v1.19.0! 🎉 Thank you all so much for your collaboration!