kops does not delete NLB load balancers

[jpds]

1. What kops version are you running? The command kops version, will display
   this information.

1.9.0-alpha1

2. What Kubernetes version are you running? kubectl version will print the
   version if a cluster is running or provide the Kubernetes version specified as
   a kops flag.

1.9.3

3. What cloud provider are you using?

AWS.

4. What commands did you run? What is the simplest way to reproduce this issue?

kops delete cluster --name=${NAME} --yes

5. What happened after the commands executed?

Kubernetes nodes were torn down by the NLB provisioned through a LoadBalancer service stayed which blocked the deletion of the internet-gateway for the VPC.

6. What did you expect to happen?

kops delete cluster deletes the NLB.

[chrislovecnm]

I think this is an open issue upstream. @micahhausler any recommendations?

[justinsb]

I think this is likely a kops issue (assuming the NLB is tagged). When kops deletes a cluster, it deletes the resources which were created by that cluster also (e.g. EBS volumes). I presume that's what you mean @jpds - you created an NLB using the magic annotation on a service of Type: LoadBalancer, but then it wasn't cleaned up when you deleted the cluster?

[baldeynz]

Hi - this also occurs using kops 1.9.0 and kubernetes 1.9.7.

kops delete cluster --name=${NAME} --yes eventually times out trying to delete the VPC because there is still nlb(s) that havent been cleaned up.

And yes @justinsb in our case the NLBs were created using the magic annotation on a service of Type: LoadBalancer.

[nareshku]

It appears that NLBs and ALBs are only supported in aws sdk package elbv2 and kops is currently using elb package as part of list and deleting load balancers.

[nareshku]

FYI, I am working on this feature. I am plugging in additional method to get ELBV2 resources and delete them just like how we do it for regular Load Balancers.

[justinsb]

Thanks @nareshku !

[nareshku]

@justinsb I am kind of stuck in here and hoping you can help me with this. I am able to get the functionality of discovering and deleting the resources working, but only with something like below:

https://github.com/nareshku/kops/blob/support-albs-nlbs/pkg/resources/aws/aws.go#L1569-L1570

svc := elbv2.New(session.New(&aws.Config{Region: aws.String(c.Region())}))
	err := svc.DescribeLoadBalancersPages(request, func(p *elbv2.DescribeLoadBalancersOutput, lastPage bool) bool {
		if len(p.LoadBalancers) == 0 {
			return true
		}

instead of

c, ok := cloud.(awsup.AWSCloud)
err := c.ELBV2().DescribeLoadBalancersPages(request, func(p *elbv2.DescribeLoadBalancersOutput, lastPage bool) bool {
		if len(p.LoadBalancers) == 0 {
			return true
		}

I am getting the below null pointer exception.

ok***** c object &{0xc4201291c0 0xc4201291e8 0xc420129218 0xc420129240 <nil> 0xc420129268 0xc420129290 us-west-2 map[KubernetesCluster:mycluster.cluster.k8s.local] 0xc4202e7f40} ************ svc object &{0xc420690600} ******

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1beed96]

goroutine 1 [running]:
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2.(*ELBV2).newRequest(0x0, 0xc420ba9200, 0x3c8f100, 0xc420ba91c0, 0x3abc820, 0xc420c5f860, 0x40)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2/service.go:87 +0x26
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2.(*ELBV2).DescribeLoadBalancersRequest(0x0, 0xc420ba91c0, 0x40, 0x3d309a0)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2/api.go:1559 +0x239
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2.(*ELBV2).DescribeLoadBalancersPagesWithContext.func1(0xc4213491e0, 0x106da01, 0x5efa460)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2/api.go:1639 +0x59
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/aws/request.(*Pagination).Next(0xc4213491e0, 0xc420ba9180)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/aws/request/request_pagination.go:92 +0x49
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2.(*ELBV2).DescribeLoadBalancersPagesWithContext(0x0, 0x7ca6eb8, 0xc4200ec008, 0xc420ba9140, 0xc420af7530, 0x0, 0x0, 0x0, 0x1, 0xc420af7530)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2/api.go:1647 +0x13b
k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2.(*ELBV2).DescribeLoadBalancersPages(0x0, 0xc420ba9140, 0xc420af7530, 0x1, 0x1)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/aws/aws-sdk-go/service/elbv2/api.go:1621 +0x90
k8s.io/kops/pkg/resources/aws.DescribeELBV2s(0x7ca6e70, 0xc420361200, 0xc420819c80, 0xc421349410, 0x18, 0xc420819c80, 0x100f38d, 0xc42003a000)
	/Users/nareshku/go/src/k8s.io/kops/pkg/resources/aws/aws.go:1572 +0x3e9
k8s.io/kops/pkg/resources/aws.ListELBV2s(0x7ca6e70, 0xc420361200, 0x7ffeefbff280, 0x18, 0xc420361200, 0x0, 0x0, 0x0, 0x0)
	/Users/nareshku/go/src/k8s.io/kops/pkg/resources/aws/aws.go:1520 +0x4d
k8s.io/kops/pkg/resources/aws.ListResourcesAWS(0x43a4440, 0xc420361200, 0x7ffeefbff280, 0x18, 0xc420361200, 0xc420e2b928, 0x102b624)
	/Users/nareshku/go/src/k8s.io/kops/pkg/resources/aws/aws.go:88 +0x2d5
k8s.io/kops/pkg/resources/ops.ListResources(0x7ca6e70, 0xc420361200, 0x7ffeefbff280, 0x18, 0x0, 0x0, 0x0, 0x6, 0xffffffffffffffff)
	/Users/nareshku/go/src/k8s.io/kops/pkg/resources/ops/collector.go:37 +0x25c
main.RunDeleteCluster(0xc420a677a0, 0x43543c0, 0xc4200fe008, 0xc420a5efc0, 0x0, 0x0)
	/Users/nareshku/go/src/k8s.io/kops/cmd/kops/delete_cluster.go:134 +0x5ec
main.NewCmdDeleteCluster.func1(0xc42041b400, 0xc420a67e20, 0x0, 0x2)
	/Users/nareshku/go/src/k8s.io/kops/cmd/kops/delete_cluster.go:79 +0xd6
k8s.io/kops/vendor/github.com/spf13/cobra.(*Command).execute(0xc42041b400, 0xc420a67de0, 0x2, 0x2, 0xc42041b400, 0xc420a67de0)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/spf13/cobra/command.go:757 +0x2c1
k8s.io/kops/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x5f098a0, 0x36b6800, 0x0, 0x0)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/spf13/cobra/command.go:843 +0x30a
k8s.io/kops/vendor/github.com/spf13/cobra.(*Command).Execute(0x5f098a0, 0x5f3e488, 0x0)
	/Users/nareshku/go/src/k8s.io/kops/vendor/github.com/spf13/cobra/command.go:791 +0x2b
main.Execute()
	/Users/nareshku/go/src/k8s.io/kops/cmd/kops/root.go:96 +0x87
main.main()
	/Users/nareshku/go/src/k8s.io/kops/cmd/kops/main.go:25 +0x20

[nareshku]

I did a print on the cloud interface and seems like elbv2iface.ELBV2API which I have included in the AWSCloud interface is nil

c: &{0xc4205237a0 0xc4205237c8 0xc4205237f8 0xc420523820 <nil> 0xc420523848 0xc420523870 us-west-2 map[KubernetesCluster:naresh.cluster.k8s.local] 0xc420411e90}

[nareshku]

never mind, I have found where to initialize session for ELBV2.