Add the ability to destroy environments through the pipeline

[kmanning]

Request
Ability to destroy resources via terraform-pipeline.

Why?
Deploying at a predetermined time can cause "release" problems. One example is listed below.

Auto Scaling activity failed 73 seconds ago with error: We currently do not have sufficient t3.medium capacity in the Availability Zone you requested (us-east-1b).

How?
AWS EC2 Capacity Reservation is requested for each AZ, X hours before the predetermined release time. After the release is complete, the AWS EC2 Capacity Reservation is cancelled.

aws_ec2_capacity_reservation
https://www.terraform.io/docs/providers/aws/r/ec2_capacity_reservation.html

[kmanning]

Hrm - can you clarify? How exactly would you expect to use this in a project?

Are you expecting the TerraformDestroyStage to accept an argument of a particular resource?

As it stands, terraform-pipeline has the concept of entire 'environments', regardless of how many or how few resources exist within that environment.

Moreover, my (possibly incorrect) assumption, would be that the UseCase for this new Stage, would simply be to retire an entire application, or environment - just once. Something like:

def destroyQa = new TerraformDestroyStage('qa')
def destroyUat = new TerraformDestroyStage('uat')
def destroyProd = new TerraformDestroyStage('prod')

destroyQa.then(destroyUat).then(destroyProd)

I'd run this once, and then delete all the source code.

^--- What I'm describing doesn't sound like what you're hoping to get.

[kmanning]

terraform

# AWS
provider "aws" {
  version = "~> 2.6"
  region  = "us-east-1"
}

locals {
  instance_count = 5
}

resource "aws_ec2_capacity_reservation" "us-east-1a" {
  instance_type     = "t3.medium"
  ebs_optimized     = true
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1a"
  instance_count    = "${local.instance_count}"
}

resource "aws_ec2_capacity_reservation" "us-east-1b" {
  instance_type     = "t3.medium"
  ebs_optimized     = true
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1b"
  instance_count    = "${local.instance_count}"
}

resource "aws_ec2_capacity_reservation" "us-east-1c" {
  instance_type     = "t3.medium"
  ebs_optimized     = true
  instance_platform = "Linux/UNIX"
  availability_zone = "us-east-1c"
  instance_count    = "${local.instance_count}"
}

terraform-pipeline

def validate = new TerraformValidateStage()
def deployQa = new TerraformEnvironmentStage('qa')
validate.then(deployQa).build()

// pause

def destroyQa = new TerraformDestroyStage('qa')
validate.then(destroyQa).build()

We would bring this stack up as needed.

[kmanning]

Cool, so:

1. It definitely sounds like you're good with TerraformDestroyStage destroying an entire environment, and NOT a specific resource in an environment.
2. When you say pause - what does that mean? Does pause represent the actual pipeline waiting for user input, and if received, it'll continue on and destroy qa? Or does pause represent a subsequent change to the Jenkinsfile, and a subsequent commit, and a subsequent pipeline run that just destroys the qa environment?
3. TerraformEnvironmentStage currently prompts the user before applying the change. I feel like it would be a good one-to-one to have TerraformDestroy prompt the user, to say something like, "Do you really want to do this?" and have to hit "Yes". Are you good with that too?
4. If you were to Apply TerraformDestroy to an environment that uses the CRQPlugin, I assume that the CRQPlugin should also open and close a CRQ before destroying the environment. Actually, ALL plugins that affect the TerraformEnvironmentStage should likely be applied to the TerraformDestroyStage.

[kmanning]

1. pause -- TerraformDestroy prompting the user would be the pause in the pipeline.
2. ALL plugins that affect the TerraformEnvironmentStage should likely be applied to the TerraformDestroyStage. Makes sense to me.

[kmanning]

Ok, that said, I think I might try to make terraform-pipeline work like this:

def validate = new TerraformValidateStage()
def deployQa = new TerraformEnvironmentStage('qa')
def destroyQa = new TerraformDestroyEnvironmentStage('qa')

validate.then(deployQa)
        .then(destroyQa)
        .build()

^-- Small distinction, but build() is intended to only be called once, and the "pipeline" is expected to be perfectly linear (I never imagined supporting anything other than a perfectly linear pipeline, at least).

Per the above, destroyQa would behave much like deployQa, possibly doing a terraform plan -destroy, and then automatically prompting the user with something along the lines of "You're about to terminate the QA environment - are you sure you want to do this?", and the user has to explicitly click "yes" in order to proceed forward.

TerraformEnvironmentStage also behaves differently in master vs a branch. For one-to-one behavior, I think I would make TerraformDestroy behave the same way too. If you're on a branch, then TerraformDestroyEnvironmentStage might run a terraform plan -destroy, but not provide the user the ability to actually destroy the environment without merging into master.\

How's that sound?

[kmanning]

Bumping to a future release

[kmanning]

UseCase: I'm ready to decommission my application, and delete/remove all my resources.

// Run  plan for destroy on qa, then  wait for Human  Confirmation
def destroyQa = new TerraformDestroyStage('qa')

// Run  plan for destroy on uat, then  wait for Human  Confirmation
def destroyUat = new TerraformDestroyStage('uat')

// Run  plan for destroy on prod, then  wait for Human  Confirmation
def destroyProd = new TerraformDestroyStage('prod')


destroyQa.then(deployUat)
         .then(deployProd)
         .build()

Behavior:

• All the same plugins should work. Eg: ParameterStorePlugin, S3BackendPluging, ChangeManagementplugins, etc, etc...
• I expect a Plan to be run, then human confirmation still expected before actually destroying. I can abort, and the destroy will not move forward.

Questions:

• Should TerraformDestroyStage just extend TerraformEnvironmentStage?
• Do we need a separate TerraformDestroyCommand? Can we reuse/adapt TerraformPlanCommand?

[kmanning]

Note:

The ConditionalApplyPlugin skips terraform apply on anything other than master by default. The equivalent behavior should probably exist for Destroy - anything other than master should not be allowed to destroy by default.

[kmanning]

On further thought, I think this issue could use some rework.

• I'm concerned about updating all plugins to decorate the new DESTROY portion of the Stage (eg: stage.decorate(DESTROY, addParameterStoreBuildWrapper(options))). I want the Stage hooks to be better defined before we extend them in this way. Already, this will break our internal ChangeManagement plugins. I don't want to discourage the use of external plugins, and extending behavior in this way will be discouraging.
• I love the idea of separating out the deployment strategies as a first-class object, but I think that too needs some refinement and definition. I don't want to release that as part of a minor release - it should go into a major release (eg: 6.0). Let's take that out for now, try to implement this feature without it as part of a v5.9, then reassess what this feature will look like under a 6.0 implementation.

[kmanning]

Have we made this feature to easy to use?

What if, as part of confirmation, we required the user to type in the name of the application environment that they're destroying? If they don't type in the right thing, we skip the destroy.

Breaking this out into a separate issue, for some future release. #261

[kmanning]

Creating a separate issue for the original UseCase - Issue #262