Update project delete to empty s3 buckets before deleting

[iamhopaul123]

Currently project delete will fail if users deployed a pipeline (basically because the s3 bucket will not be empty and throw an error when trying to delete them). We should clear all objects in s3 buckets before we try to delete them.

[gabelton]

Hi @iamhopaul123 , just wondering if it's possible to achieve something similar when running copilot env delete? Currently the addons stack will fail to delete if the s3 bucket isn't empty, even if DeletionPolicy is set to delete

[iamhopaul123]

Hello @gabelton. It's definitely not expected to run into this deletion issue. However, for the s3 bucket did you mean the s3 bucket in our addons? Or it's the same bucket Copilot creates through the stackset instance?

[gabelton]

Thanks for the quick reply @iamhopaul123 . I mean a bucket we create in our addons. The repo folder structure is copilot/environments/addons/demodjango-s3-bucket.yml and the file looks like:

Parameters:
  # Copilot required Parameters...
  App:
    Type: String
    Description: Your application's name.
  Env:
    Type: String
    Description: The environment name your service, job, or workflow is being deployed to.

Mappings:
  demodjangoS3BucketEnvironmentConfigMap:
    # Create an entry for each environment

    ant:
      BucketName: 'demodjango-s3-bucket-ant'
      DeletionPolicy: Delete

    staging:
      BucketName: 'demodjango-s3-bucket-staging'
      DeletionPolicy: Delete

    jayesh:
      BucketName: 'demodjango-s3-bucket-jayesh'
      DeletionPolicy: Delete

    willg:
      BucketName: 'demodjango-s3-bucket-willg'
      DeletionPolicy: Delete

    dev:
      BucketName: 'demodjango-s3-bucket-dev'
      DeletionPolicy: Delete

    john:
      BucketName: 'demodjango-s3-bucket-john'
      DeletionPolicy: Delete

    yusuf:
      BucketName: 'demodjango-s3-bucket-yusuf'
      DeletionPolicy: Delete

    gabe:
      BucketName: 'demodjango-s3-bucket-gabe'
      DeletionPolicy: Delete


Resources:
  demodjangoS3BucketKMSKey:
    Type: "AWS::KMS::Key"
    DeletionPolicy: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, DeletionPolicy]
    UpdateReplacePolicy: Retain
    Properties:
      Description: "KMS Key for RDS encryption"
      KeyPolicy:
        Version: '2012-10-17'
        Id: !Sub '${App}-${Env}-demodjangoS3Bucket-key'
        Statement:
        - Sid: Enable IAM User Permissions
          Effect: Allow
          Principal:
            AWS: !Sub "arn:aws:iam::${AWS::AccountId}:root"
          Action: kms:*
          Resource: '*'

  demodjangoS3BucketKeyAlias:
    Type: 'AWS::KMS::Alias'
    DeletionPolicy: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, DeletionPolicy]
    UpdateReplacePolicy: Retain
    Properties:
      AliasName: !Sub 'alias/${App}-${Env}-demodjangoS3Bucket-key'
      TargetKeyId: !Ref demodjangoS3BucketKMSKey

  demodjangoS3BucketBucket:
    Metadata:
      'aws:copilot:description': 'An Amazon S3 bucket to store and retrieve objects for demodjangoS3Bucket'
    Type: AWS::S3::Bucket
    DeletionPolicy: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, DeletionPolicy]
    UpdateReplacePolicy: Retain
    Properties:
      BucketName: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, BucketName]
      AccessControl: Private
      BucketEncryption:
        ServerSideEncryptionConfiguration:
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms
            KMSMasterKeyID: !Ref demodjangoS3BucketKMSKey
      VersioningConfiguration:
        Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      Tags:
        - Key: Copilot-application
          Value: !Ref App
        - Key: Copilot-environment
          Value: !Ref Env
        - Key: Copilot-deletion-policy
          Value: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, DeletionPolicy]

  demodjangoS3BucketBucketPolicy:
    Metadata:
      'aws:copilot:description': 'A bucket policy to deny unencrypted access to the bucket and its contents'
    Type: AWS::S3::BucketPolicy
    DeletionPolicy: !FindInMap [demodjangoS3BucketEnvironmentConfigMap, !Ref Env, DeletionPolicy]
    UpdateReplacePolicy: Retain
    Properties:
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Sid: ForceHTTPS
            Effect: Deny
            Principal: '*'
            Action: 's3:*'
            Resource:
              - !Sub ${ demodjangoS3BucketBucket.Arn}/*
              - !Sub ${ demodjangoS3BucketBucket.Arn}
            Condition:
              Bool:
                "aws:SecureTransport": false
      Bucket: !Ref demodjangoS3BucketBucket



Outputs:
  demodjangoS3BucketName:
    Description: "The name of a user-defined bucket."
    Value: !Ref demodjangoS3BucketBucket

  demodjangoS3BucketKMSKeyARN:
    Description: "ARN of the KMS Key"
    Value: !GetAtt demodjangoS3BucketKMSKey.Arn
    Export:
      Name: !Sub "${App}-${Env}-demodjangoS3Bucket-KMSKeyARN"

[iamhopaul123]

Gotcha. Sorry I don't think Copilot automatically does that for you because addons resources are defined and managed by users. You will have to manually emptying the bucket before deleting it entirely.

Currently the addons stack will fail to delete if the s3 bucket isn't empty, even if DeletionPolicy is set to delete

However, this is unexpected and surprising to me. Then maybe a custom resource that empties the bucket before bucket deletion might be your best bet https://stackoverflow.com/questions/40383470/can-i-force-cloudformation-to-delete-non-empty-s3-bucket

[gabelton]

Thanks @iamhopaul123 . I was looking at custom resources / lambda as a solution, but was hoping to avoid it, if there was something out-of-the-box that I was missing. Doesn't look like there is though