Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to delete hosted zone with DNS verified certificate #7063

Closed
strottos opened this issue Mar 29, 2020 · 34 comments · Fixed by #18311
Closed

Unable to delete hosted zone with DNS verified certificate #7063

strottos opened this issue Mar 29, 2020 · 34 comments · Fixed by #18311
Labels
@aws-cdk/aws-route53 Related to Amazon Route 53 bug This issue is a bug. effort/medium Medium work item – several days of effort p1

Comments

@strottos
Copy link

strottos commented Mar 29, 2020

When I try to delete a stack that has a hosted zone with a certificate created through certificate manager, it has a CNAME entry created by certificate manager that prevents the hosted zone being deleted. Error report attached below.

My case is as follows though it's possible you don't need separate accounts, I haven't checked. I have two accounts, one that has the DNS hosted zone and name servers for my existing domain name (say Account A) and a separate one that I'm creating a DNS hosted zone underneath (say Account B). For example, account A has the Route53 records for my domain example.com and I want to be able to create a subdomain test.example.com in account B that has a valid certificate.

I've got 3 CDK stacks do this, one to create the hosted zone in Account B, one to add the nameservers from that account to Account A and finally one to create the certificate in Account B (see the code samples below). This all works really well for creation but for deletion the certificate manager stack has left a CNAME entry similar to the following in the hosted zone preventing deletion:

_72fb0....bc94c.test.example.com. | CNAME | _c1c3d...ebd76.vhzmpjdqfx.acm-validations.aws. |  
-- | -- | -- | --

I would have expected the certificate manager that created this to remove it as well.

Reproduction Steps

Release the following stacks in order and delete them in reverse order, you'll need to pass in appropriate environment variables.

Stack 1:

import os

from aws_cdk import aws_route53 as route53, core


class DnsHostedZoneStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        domain_url = os.environ["domain_url"]

        route53.PublicHostedZone(self, "DomainEnvironment", zone_name=domain_url)

Stack 2:

import json
import os

from aws_cdk import aws_route53 as route53, core


class DnsLinkHostedZoneStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        root_domain_url = os.environ["root_domain_url"]
        environment_subdomain = os.environ["environment_subdomain"]
        hosted_zone_id = os.environ["hosted_zone_id"]
        nameservers = json.loads(os.environ["nameservers"])

        full_domain_url = environment_subdomain + "." + root_domain_url

        main_hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "MainHostLookup",
            zone_name=root_domain_url,
            hosted_zone_id=hosted_zone_id,
        )

        route53.RecordSet(
            self,
            "NameServerLink",
            record_name=full_domain_url,
            record_type=route53.RecordType.NS,
            zone=main_hosted_zone,
            target=route53.RecordTarget.from_values(*nameservers),
        )

Stack 3:

import os

from aws_cdk import (
    aws_route53 as route53,
    aws_certificatemanager as certificatemanager,
    core,
)


class CertificateStack(core.Stack):
    def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
        super().__init__(scope, id, **kwargs)

        hosted_domain_url = os.environ["hosted_domain_url"]
        hosted_zone_id = os.environ["hosted_zone_id"]
        hosted_zone = route53.HostedZone.from_hosted_zone_attributes(
            self,
            "MainHostLookup",
            zone_name=hosted_domain_url,
            hosted_zone_id=hosted_zone_id,
        )

        self.certificate = certificatemanager.DnsValidatedCertificate(
            self,
            "EnvironmentCertificate",
            hosted_zone=hosted_zone,
            domain_name=hosted_domain_url,
            region="us-east-1",
            subject_alternative_names=[f"*.{hosted_domain_url}"],
            validation_method=certificatemanager.ValidationMethod.DNS,
        )

Error Log

   0 | 5:07:47 PM | DELETE_IN_PROGRESS   | AWS::CDK::Metadata       | CDKMetadata
   1 | 5:07:48 PM | DELETE_FAILED        | AWS::Route53::HostedZone | DomainEnvironment (DomainEnvironment7F28B06E) The specified hosted zone contains non-required resource record sets  and so cannot be deleted. (Service: AmazonRoute53; Status Code: 400; Error Code: HostedZoneNotEmpty; Request ID: 33c379cc-bede-48e8-8083-205a44fed498)
        new HostedZone (/private/tmp/jsii-kernel-HYp2Da/node_modules/@aws-cdk/aws-route53/lib/hosted-zone.js:16:26)
        \_ new PublicHostedZone (/private/tmp/jsii-kernel-HYp2Da/node_modules/@aws-cdk/aws-route53/lib/hosted-zone.js:116:9)
        \_ /Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7838:49
        \_ Kernel._wrapSandboxCode (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:8298:20)
        \_ Kernel._create (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7838:26)
        \_ Kernel.create (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7585:21)
        \_ KernelHost.processRequest (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7372:28)
        \_ KernelHost.run (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7312:14)
        \_ Immediate._onImmediate (/Users/<snip>/lib/python3.6/site-packages/jsii/_embedded/jsii/jsii-runtime.js:7315:37)
        \_ processImmediate (internal/timers.js:439:21)
   2 | 5:07:48 PM | DELETE_COMPLETE      | AWS::CDK::Metadata       | CDKMetadata
   3 | 5:07:48 PM | DELETE_FAILED        | AWS::CloudFormation::Stack | dns-hosted-zone The following resource(s) failed to delete: [DomainEnvironment7F28B06E].

 ❌  dns-hosted-zone: destroy failed Error: The stack named dns-hosted-zone is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [DomainEnvironment7F28B06E]. )
    at /Users/<snip>/node_modules/aws-cdk/lib/api/util/cloudformation.ts:165:13
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at waitFor (/Users/<snip>/node_modules/aws-cdk/lib/api/util/cloudformation.ts:76:20)
    at Object.destroyStack (/Users/<snip>/node_modules/aws-cdk/lib/api/deploy-stack.ts:261:26)
    at CdkToolkit.destroy (/Users/<snip>/node_modules/aws-cdk/lib/cdk-toolkit.ts:211:9)
    at main (/Users/<snip>/node_modules/aws-cdk/bin/cdk.ts:245:16)
    at initCommandLine (/Users/<snip>/node_modules/aws-cdk/bin/cdk.ts:172:9)
The stack named dns-hosted-zone is in a failed state: DELETE_FAILED (The following resource(s) failed to delete: [DomainEnvironment7F28B06E]. )

Environment

  • CLI Version : 1.31
  • Framework Version: 1.31 (guessing this is the Python library versions)
  • OS : Mac
  • Language : Python

Other


This is 🐛 Bug Report

@strottos strottos added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Mar 29, 2020
@SomayaB SomayaB added the @aws-cdk/aws-route53 Related to Amazon Route 53 label Mar 30, 2020
@shivlaks shivlaks added the p2 label Apr 2, 2020
@shivlaks shivlaks added the effort/medium Medium work item – several days of effort label Apr 13, 2020
@ayush987goyal
Copy link
Contributor

Encountered the same issue myself. The lambda creates a CNAME record set in the hosted zone but does not delete the same while deleting the certificate.

Any updates on this or how are we thinking of handling it?

@SomayaB SomayaB removed the needs-triage This issue or PR still needs to be triaged. label May 19, 2020
@strottos
Copy link
Author

Possible this may be fixed by the following PR given that this will build this as native CloudFormation (though may need verifying once that has merged): #8552

@moltar
Copy link
Contributor

moltar commented Aug 17, 2020

Just ran into the same issue on 1.56.0.

Any suggested workarounds for now?

@chasemaier
Copy link

Still present in 1.61.0. It would be nice if the hosted zone was created as part of the stack if it could be flagged as "okay to delete all records sets upon zone deletion". Perhaps that's a Cloudformation limitation though?

It wouldn't be so bad, but the stack rollback eliminates anything but the hosted zone and then fails to complete the rollback. You can't then simply redeploy the stack since that creates a new zone (with the same problem), and you have to swap around NS records with each attempt to deploy a zone followed by some manual cleanup of the DNS validation records for the certificates.

@shivlaks
Copy link
Contributor

@moltar @chasemaier - marking this as a p1 so that it gets prioritized. I'll need to work through the repro steps to start putting together a fix. will post an update when I have more details to share. stay tuned!

@shivlaks shivlaks added p1 and removed p2 labels Aug 28, 2020
@moltar
Copy link
Contributor

moltar commented Aug 28, 2020

I can make a repro repo example, if that'd help.

@shivlaks
Copy link
Contributor

@moltar it would help! any minimal repro would expedite triage resolution :)

@strottos
Copy link
Author

It occurs to me this could well be a CloudFormation limitation but that doesn't mean CDK can't do one better maybe (though I don't know). It seems that when you manually delete the CloudFormation stack that created the cert it still leaves this CNAME entry in place, which is what causes the problem. Maybe CDK can recognise that if such a CNAME as would have been created still exists it deletes it, or at least can be pushed forward to a CloudFormation or beyond team if the problem lays there.

@moltar
Copy link
Contributor

moltar commented Aug 29, 2020

Here you go: https://github.com/moltar/cdk-hosted-zone

I also added repro steps in the README

@Dzhuneyt
Copy link
Contributor

Dzhuneyt commented Oct 6, 2020

I was thinking of maybe attaching a Lambda "Custom Resource" that does a "cleanup" of DNS records within the hosted zone as part of the HostedZone deletion. It seems kind of risky though? What's the opinion of CDK maintainers on this topic?

This approach is very similar to the s3.Bucket being often replaced by AutoDeleteBucket. Maybe we (the community) should develop a third party AutoDeleteHostedZone construct?

@strottos
Copy link
Author

strottos commented Oct 12, 2020

Seems to me that the bug is in the ACM teardown. ACM setup that CNAME entry in the first place, when it's deleted it should do the cleanup too. I'd still expect the DNS hosted zone to fail deletion if I've manually added a record that wasn't cleaned up say. In the same way that with an S3 bucket I'd expect it to not clearup if there's still data in it.

@moltar
Copy link
Contributor

moltar commented Nov 23, 2020

This is extremely frustrating in CI, because the cleanup cannot happen automatically.

@zacklitzsinger
Copy link

I was able to get around this issue by writing a script that automatically removes the generated CNAME record from a given hosted zone.

https://gist.github.com/zacklitzsinger/26051489d05aca19e3c360324028717d

@mikestopcontinues
Copy link

I just want to highlight @Dzhuneyt: While the verification record should definitely be auto-deleted, we really need an option to delete all records when tearing down the hosted zone. This issue comes up often.

@stevehodgkiss
Copy link
Contributor

Similar to @zacklitzsinger, we implemented a custom resource lambda that automatically deletes hosted zone records when the stack is being deleted. We only turn this on in CI/test/development environments though. https://gist.github.com/stevehodgkiss/01241f0ca57e22557f592ee87dc667f9

While the verification record should definitely be auto-deleted

I'm not sure that's true. AFAIK automatically cleaning up these entries after validation might remove records for other certificates and break renewals for them. I added some references to relevant PRs when I implemented this ~2 months ago. https://gist.github.com/stevehodgkiss/01241f0ca57e22557f592ee87dc667f9#file-stack-usage-ts-L4

Having this option in HostedZone itself would be ideal 👍

boonjiashen added a commit to boonjiashen/boonjiashen-dot-com-infra that referenced this issue Sep 6, 2021
We used this for the ElasticBeanstalk app which I've deleted

__Testing done__

1. `cdk deploy infraStackProd infraStackBeta`
1. See certifate gone from cfn console
1. Needed to manually delete the CNAME record from the hosted zone, which CDK doesn't manage. See aws/aws-cdk#7063
@kierans
Copy link

kierans commented Sep 22, 2021

As a newcomer to this issue, I agree with @strottos that if the ACM setup adds the CNAME, it should delete it. Hopefully this issue can get fixed soon.

njlynch added a commit that referenced this issue Jan 7, 2022
Adds an option to DnsValidatedCertificate to automatically cleanup the related
DNS validation records when the Certificate is deleted.

This is an opt-in property and discouraged for production use, as there are edge
cases that can cause unintended side effects. The most obvious is that if two or
more certificates exist with the same domain, the same validation record is used
for both. If one certificate is deleted (and deletes the validation record), the
second certificate (with the same domain name) will be unable to automatically
renew.

closes #3333
closes #7063
njlynch added a commit that referenced this issue Jan 7, 2022
Adds an option to DnsValidatedCertificate to automatically cleanup the related
DNS validation records when the Certificate is deleted.

This is an opt-in property and discouraged for production use, as there are edge
cases that can cause unintended side effects. The most obvious is that if two or
more certificates exist with the same domain, the same validation record is used
for both. If one certificate is deleted (and deletes the validation record), the
second certificate (with the same domain name) will be unable to automatically
renew.

closes #3333
closes #7063
@mergify mergify bot closed this as completed in #18311 Jan 25, 2022
mergify bot pushed a commit that referenced this issue Jan 25, 2022
…18311)

Adds an option to DnsValidatedCertificate to automatically cleanup the related
DNS validation records when the Certificate is deleted.

This is an opt-in property and discouraged for production use, as there are edge
cases that can cause unintended side effects. The most obvious is that if two or
more certificates exist with the same domain, the same validation record is used
for both. If one certificate is deleted (and deletes the validation record), the
second certificate (with the same domain name) will be unable to automatically
renew.

closes #3333
closes #7063

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

LukvonStrom pushed a commit to LukvonStrom/aws-cdk that referenced this issue Jan 26, 2022
…ws#18311)

Adds an option to DnsValidatedCertificate to automatically cleanup the related
DNS validation records when the Certificate is deleted.

This is an opt-in property and discouraged for production use, as there are edge
cases that can cause unintended side effects. The most obvious is that if two or
more certificates exist with the same domain, the same validation record is used
for both. If one certificate is deleted (and deletes the validation record), the
second certificate (with the same domain name) will be unable to automatically
renew.

closes aws#3333
closes aws#7063

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this issue Feb 21, 2022
…ws#18311)

Adds an option to DnsValidatedCertificate to automatically cleanup the related
DNS validation records when the Certificate is deleted.

This is an opt-in property and discouraged for production use, as there are edge
cases that can cause unintended side effects. The most obvious is that if two or
more certificates exist with the same domain, the same validation record is used
for both. If one certificate is deleted (and deletes the validation record), the
second certificate (with the same domain name) will be unable to automatically
renew.

closes aws#3333
closes aws#7063

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@skkrail-amzn
Copy link

I see this fix was added in v1.141.0, but I'm still experiencing this issue on v1.152.0. Any suggestions?

@pandieme
Copy link

pandieme commented May 29, 2022

I see this fix was added in v1.141.0, but I'm still experiencing this issue on v1.152.0. Any suggestions?

@skkrail-amzn I've run into this myself and come across this thread. On your certificate aws-certificatemanager.DnsValidatedCertificate object, you would need to specifically set the cleanupRoute53Records boolean to true as it's an opt-in feature added in #18311.

E.g.

const certificate = new acm.DnsValidatedCertificate(this, "SiteCertificate", {
      domainName: props.domainName,
      hostedZone: props.zone,
      region: "us-east-1", // Cloudfront only checks this region for certificates.
      cleanupRoute53Records: true
    });

@skkrail-amzn
Copy link

Thanks @runtooctober! I was using aws-certificatemanager.Certificate with the validation prop set to CertificateValidation.fromDns(publicHostedZone). Switching to aws-certificatemanager.DnsValidatedCertificate like you mentioned gave me access to this feature.

@wv-tud
Copy link

wv-tud commented Feb 15, 2023

Ironically aws-certificatemanager.DnsValidatedCertificate is being deprecated and the recommended alternative is aws-certificatemanager.Certificate which exhibits the same issue for me when using DNS validation.

@petro2050
Copy link

There should be a similar cleanupRoute53Records property in certificatemanager.Certificate if the other construct is deprecated. It's frustrating that this issue hasn't been fixed in over 3 years.

@koldova
Copy link

koldova commented Jul 2, 2023

Are there any workaround or update on this?

@suharsha
Copy link

suharsha commented Jul 7, 2023

Having the same issue with Certificate which is frustrating. Any fixes please?

@evgenyka
Copy link
Contributor

Having the same issue with Certificate which is frustrating. Any fixes please?

This issue is closed as fixed by #18311. Can you please check if the referred fix covers your problem?

@brubraz
Copy link

brubraz commented Jul 18, 2023

Having the same issue with Certificate which is frustrating. Any fixes please?

This issue is closed as fixed by #18311. Can you please check if the referred fix covers your problem?

It appears that the current solution only works for the deprecated DnsValidatedCertificate. It would be great to include this feature for Certificate as well.

@dguisinger
Copy link

+1 .... This issue is still occurring, please fix.

@buryo
Copy link

buryo commented Nov 7, 2023

Facing the same issue using CloudFormation... Definitely needs a fix

@paul-uz
Copy link

paul-uz commented Nov 16, 2023

Experiencing this issue in CF as well. Certificate created in CF yml, and added to a HostedZone, also created in the same yml. If the stack fails to update and has to rollback, the HostedZone cannot be deleted due to the validation CNAME record

@gardicam
Copy link

Experiencing this as well

@MyNameIsOka
Copy link

This is quite frustrating. Would highly appreciate a fix.

@thpham
Copy link

thpham commented Apr 19, 2024

Similar to @zacklitzsinger, we implemented a custom resource lambda that automatically deletes hosted zone records when the stack is being deleted. We only turn this on in CI/test/development environments though. https://gist.github.com/stevehodgkiss/01241f0ca57e22557f592ee87dc667f9

While the verification record should definitely be auto-deleted

I'm not sure that's true. AFAIK automatically cleaning up these entries after validation might remove records for other certificates and break renewals for them. I added some references to relevant PRs when I implemented this ~2 months ago. https://gist.github.com/stevehodgkiss/01241f0ca57e22557f592ee87dc667f9#file-stack-usage-ts-L4

Having this option in HostedZone itself would be ideal 👍

@stevehodgkiss do you still have the example you shared, the links are no more accessible, it would be very much appreciated ! thx

@tgjohnst
Copy link

Also reporting in to say I'm experiencing this issue in CF and it's getting in the way of automating stack teardown. Would really appreciate a fix along the lines of the cleanupRoute53Records parameter mentioned above in DNSValidatedCertificate, but within CF AWS::CertificateManager::Certificate

@frjonsen
Copy link

Any update on this? We have also been using a custom resource for this, but there's a bunch of edge cases when you have multiple certificates with the same domain name, and it gets trickier when having multiple SANs. This really should be built in. Someone clearly thought it should, since it was a part of the deprecated DnsValidatedConstruct.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-route53 Related to Amazon Route 53 bug This issue is a bug. effort/medium Medium work item – several days of effort p1
Projects
None yet
Development

Successfully merging a pull request may close this issue.