Skip to content
This repository has been archived by the owner on Jun 23, 2021. It is now read-only.

AWS tasks must always tag the resources they create (ec2_vpc_nat_gateway) #56

Open
johnsimcall opened this issue Aug 19, 2020 · 0 comments
Open

AWS tasks must always tag the resources they create (ec2_vpc_nat_gateway) #56

johnsimcall opened this issue Aug 19, 2020 · 0 comments
Assignees
@johnsimcall
Labels
bug Something isn't working

Comments

@johnsimcall
Copy link
Collaborator

The ec2_vpc_nat_gateway module doesn't allow tags to be created. The workaround is to immediately run a second ec2_tag task to apply the OpenShiftCluster and OpenShiftClusterId tags. The aws_destroy.yml playbook needs those tags to do it's job.

This workaround is problematic though because we create multiple NAT Gateways (one per AZ) and we aren't guaranteed to have all of them be created successfully. If any NGW failed to get created --for any reason-- the ec2_tag task would not be run. This results in creating untagged NGWs that can't be deleted by the aws_destroy.yml playbook. When the NGWs aren't deleted, then the subnets they belong to and ultimately the VPC can't be deleted.

Here is an example of a partial failure caused by an insufficient EIP Allocation quota. Only one of the three NGWs was created successfully. The logs also show the results of the patch I've created to resolve the issue. The patch simply wraps the ec2_vpc_nat_gateway task in an Ansible Block with an always: section.

Thanks to Mike for reporting the issue originally. And thanks to Gerald for suggesting the solution!

TASK [infrastructure : Create NAT gateways] ******************************************************************************************************************
Wednesday 19 August 2020  05:32:34 +0000 (0:00:02.146)       0:00:19.863 ****** 
changed: [localhost] => (item=subnet-0bdde190730995f57) => {
    "ansible_loop_var": "item",
    "changed": true,
    "create_time": "2020-08-19T05:32:36+00:00",
    "item": "subnet-0bdde190730995f57",
    "nat_gateway_addresses": [
        {
            "allocation_id": "eipalloc-08f5b15b12fda58d1"
        }
    ],
    "nat_gateway_id": "nat-0e52ccf6f78cd7bfd",
    "state": "pending",
    "subnet_id": "subnet-0bdde190730995f57",
    "success": true,
    "vpc_id": "vpc-0ae3aed1910f59a4f"
}
failed: [localhost] (item=subnet-029f25413093ecfe2) => {
    "ansible_loop_var": "item",
    "changed": "False",
    "item": "subnet-029f25413093ecfe2",
    "success": false
}

MSG:

An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.
failed: [localhost] (item=subnet-06c05d404fea71bf2) => {
    "ansible_loop_var": "item",
    "changed": "False",
    "item": "subnet-06c05d404fea71bf2",
    "success": false
}

MSG:

An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.

TASK [infrastructure : Add NAT gateway tags] *****************************************************************************************************************
Wednesday 19 August 2020  05:32:38 +0000 (0:00:03.753)       0:00:23.617 ****** 
changed: [localhost] => (item=[{'msg': '', 'success': True, 'changed': True, 'create_time': '2020-08-19T05:32:36+00:00', 'nat_gateway_addresses': [{'allocation_id': 'eipalloc-08f5b15b12fda58d1'}], 'nat_gateway_id': 'nat-0e52ccf6f78cd7bfd', 'state': 'pending', 'subnet_id': 'subnet-0bdde190730995f57', 'vpc_id': 'vpc-0ae3aed1910f59a4f', 'invocation': {'module_args': {'subnet_id': 'subnet-0bdde190730995f57', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'failed': False, 'item': 'subnet-0bdde190730995f57', 'ansible_loop_var': 'item'}, 'us-gov-east-1a']) => {
    "added_tags": {
        "Name": "ocp4-jcall-redhatgov-io-us-gov-east-1a",
        "OpenShiftCluster": "ocp4.jcall.redhatgov.io",
        "OpenShiftClusterId": "ocp4-jcall-redhatgov-io"
    },
    "ansible_loop_var": "item",
    "changed": true,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": true,
            "create_time": "2020-08-19T05:32:36+00:00",
            "failed": false,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-0bdde190730995f57",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-0bdde190730995f57",
            "msg": "",
            "nat_gateway_addresses": [
                {
                    "allocation_id": "eipalloc-08f5b15b12fda58d1"
                }
            ],
            "nat_gateway_id": "nat-0e52ccf6f78cd7bfd",
            "state": "pending",
            "subnet_id": "subnet-0bdde190730995f57",
            "success": true,
            "vpc_id": "vpc-0ae3aed1910f59a4f"
        },
        "us-gov-east-1a"
    ],
    "tags": {
        "Name": "ocp4-jcall-redhatgov-io-us-gov-east-1a",
        "OpenShiftCluster": "ocp4.jcall.redhatgov.io",
        "OpenShiftClusterId": "ocp4-jcall-redhatgov-io"
    }
}
skipping: [localhost] => (item=[{'msg': 'An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.', 'success': False, 'changed': 'False', 'failed': True, 'invocation': {'module_args': {'subnet_id': 'subnet-029f25413093ecfe2', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'item': 'subnet-029f25413093ecfe2', 'ansible_loop_var': 'item'}, 'us-gov-east-1b'])  => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": "False",
            "failed": true,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-029f25413093ecfe2",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-029f25413093ecfe2",
            "msg": "An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.",
            "success": false
        },
        "us-gov-east-1b"
    ],
    "skip_reason": "Conditional result was False"
}
skipping: [localhost] => (item=[{'msg': 'An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.', 'success': False, 'changed': 'False', 'failed': True, 'invocation': {'module_args': {'subnet_id': 'subnet-06c05d404fea71bf2', 'if_exist_do_not_create': True, 'debug_botocore_endpoint_logs': False, 'validate_certs': True, 'state': 'present', 'wait': False, 'wait_timeout': 320, 'release_eip': False, 'ec2_url': None, 'aws_secret_key': None, 'aws_access_key': None, 'security_token': None, 'profile': None, 'region': None, 'eip_address': None, 'allocation_id': None, 'nat_gateway_id': None, 'client_token': None}}, 'item': 'subnet-06c05d404fea71bf2', 'ansible_loop_var': 'item'}, 'us-gov-east-1c'])  => {
    "ansible_loop_var": "item",
    "changed": false,
    "item": [
        {
            "ansible_loop_var": "item",
            "changed": "False",
            "failed": true,
            "invocation": {
                "module_args": {
                    "allocation_id": null,
                    "aws_access_key": null,
                    "aws_secret_key": null,
                    "client_token": null,
                    "debug_botocore_endpoint_logs": false,
                    "ec2_url": null,
                    "eip_address": null,
                    "if_exist_do_not_create": true,
                    "nat_gateway_id": null,
                    "profile": null,
                    "region": null,
                    "release_eip": false,
                    "security_token": null,
                    "state": "present",
                    "subnet_id": "subnet-06c05d404fea71bf2",
                    "validate_certs": true,
                    "wait": false,
                    "wait_timeout": 320
                }
            },
            "item": "subnet-06c05d404fea71bf2",
            "msg": "An error occurred (AddressLimitExceeded) when calling the AllocateAddress operation: The maximum number of addresses has been reached.",
            "success": false
        },
        "us-gov-east-1c"
    ],
    "skip_reason": "Conditional result was False"
}

PLAY RECAP ***************************************************************************************************************************************************
localhost                  : ok=16   changed=1    unreachable=0    failed=1    skipped=20   rescued=0    ignored=0   

Wednesday 19 August 2020  05:32:40 +0000 (0:00:01.561)       0:00:25.178 ****** 
=============================================================================== 
Playbook run took 0 days, 0 hours, 0 minutes, 25 seconds

root at openshift4-bundle in /app 
#
@johnsimcall johnsimcall self-assigned this Aug 19, 2020
@johnsimcall johnsimcall added the bug Something isn't working label Aug 19, 2020
johnsimcall added a commit to johnsimcall/openshift4-deploy that referenced this issue Aug 19, 2020
@johnsimcall
always tag NAT Gateways - resolves jaredhocutt#56
9d27ae5
@johnsimcall johnsimcall mentioned this issue Aug 19, 2020
Open
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@johnsimcall johnsimcall

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@johnsimcall