Openstack delete dynamic floating ip in delete cluster #7045

drekle · 2019-05-21T13:21:27Z

Instead of using the cluster name to search for the appropriate loadbalancers, which limits this to the api loadbalancer provisioned by kops, I have updated this to search for loadbalancers on the subnets which kops provisions which will clean up the resources created by LoadBalancer services internal to the cluster.

Without the change kops delete cluster will fail attempting to delete a subnet with associated ports.

/sig openstack

drekle · 2019-05-21T13:26:41Z

pkg/resources/openstack/lb.go

 	var resourceTrackers []*resources.Resource
+
+	if os.osCloud.UseOctavia() {


@zetaab I am not overly familiar with octavia if you could give this a look.

if using octavia we do not need delete any subresources (listener,monitor,pool,members). Instead we can just delete loadbalancer itself and use cascade option there. So most part of the code is useless if using octavia. Example when using octavia https://github.com/kubernetes/kops/blob/master/pkg/resources/openstack/lb.go#L53-L56

Thanks ill update to reflect this.

drekle · 2019-05-21T13:34:15Z

/retest

zetaab · 2019-05-21T14:44:47Z

pkg/resources/openstack/lb.go

 	var resourceTrackers []*resources.Resource
+
+	if os.osCloud.UseOctavia() {


remove this whole if structure

pkg/resources/openstack/lb.go

drekle · 2019-05-22T18:48:57Z

/hold

…cally added ones as well, allowing for subnet deletion Bazel updates Updating deletion logic for octavia clusters

drekle · 2019-05-23T20:11:18Z

There may be some ports associated to the loadbalancer which will not be cleaned up.

zetaab

% openstack loadbalancer list
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                                                                  | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| 083287cc-c449-47df-93af-3aa871e571bd | api.testlb.k8s.local                                                  | 9ebaa4069d4d4e009a94e85a811ea8a1 | 10.1.32.13  | ACTIVE              | octavia  |
| 45d8fef4-7ccb-4167-8369-2e7841a1106a | kube_service_kubernetes_ingress-nginx-external_ingress-nginx-external | 9ebaa4069d4d4e009a94e85a811ea8a1 | 10.1.32.7   | ACTIVE              | octavia  |
| 0c998bb7-b15e-434b-b062-56a3e5a2b5a0 | kube_service_kubernetes_ingress-nginx-internal_ingress-nginx-internal | 9ebaa4069d4d4e009a94e85a811ea8a1 | 10.1.32.11  | ACTIVE              | octavia  |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+

# trying to delete cluster with current master version:
% kops delete cluster testlb.k8s.local --yes

....

I0526 12:51:57.043185   56063 context.go:231] hit maximum retries 5 with error error deleting subnet: Expected HTTP response code [] when accessing [DELETE https://helpa1.company.com13696/v2.0/subnets/29bda375-0dad-450c-b37c-6e91cdf34da6], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet 29bda375-0dad-450c-b37c-6e91cdf34da6: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}}
Subnet:29bda375-0dad-450c-b37c-6e91cdf34da6	error deleting resources, will retry: error deleting subnet: Expected HTTP response code [] when accessing [DELETE https://helpa1.company.com:13696/v2.0/subnets/29bda375-0dad-450c-b37c-6e91cdf34da6], but got 409 instead
{"NeutronError": {"message": "Unable to complete operation on subnet 29bda375-0dad-450c-b37c-6e91cdf34da6: One or more ports have an IP allocation from this subnet.", "type": "SubnetInUse", "detail": ""}}
I0526 12:51:57.478440   56063 context.go:231] hit maximum retries 5 with error error deleting network: Expected HTTP response code [] when accessing [DELETE https://helpa1.company.com:13696/v2.0/networks/4e0541b8-c7dc-4209-8f5a-45250a7ebab7], but got 409 instead

...

% openstack loadbalancer list
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| id                                   | name                                                                  | project_id                       | vip_address | provisioning_status | provider |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+
| 45d8fef4-7ccb-4167-8369-2e7841a1106a | kube_service_kubernetes_ingress-nginx-external_ingress-nginx-external | 9ebaa4069d4d4e009a94e85a811ea8a1 | 10.1.32.7   | ACTIVE              | octavia  |
| 0c998bb7-b15e-434b-b062-56a3e5a2b5a0 | kube_service_kubernetes_ingress-nginx-internal_ingress-nginx-internal | 9ebaa4069d4d4e009a94e85a811ea8a1 | 10.1.32.11  | ACTIVE              | octavia  |
+--------------------------------------+-----------------------------------------------------------------------+----------------------------------+-------------+---------------------+----------+

# Trying with this PR version
% kops delete cluster testlb.k8s.local --yes
I0526 12:57:25.275267   56879 s3context.go:87] Found S3_ENDPOINT="https://s3.company.com", using as non-AWS S3 backend
TYPE		NAME									ID
LoadBalancer	kube_service_kubernetes_ingress-nginx-external_ingress-nginx-external	45d8fef4-7ccb-4167-8369-2e7841a1106a
LoadBalancer	kube_service_kubernetes_ingress-nginx-internal_ingress-nginx-internal	0c998bb7-b15e-434b-b062-56a3e5a2b5a0
Network		testlb.k8s.local							4e0541b8-c7dc-4209-8f5a-45250a7ebab7
Subnet		zone-1.testlb.k8s.local							29bda375-0dad-450c-b37c-6e91cdf34da6

LoadBalancer:45d8fef4-7ccb-4167-8369-2e7841a1106a	ok
LoadBalancer:0c998bb7-b15e-434b-b062-56a3e5a2b5a0	ok
Subnet:29bda375-0dad-450c-b37c-6e91cdf34da6	ok
Network:4e0541b8-c7dc-4209-8f5a-45250a7ebab7	ok
Deleted kubectl config for testlb.k8s.local

Deleted cluster: "testlb.k8s.local"

This works as should if we are using octavia. @drekle if you can fix that neutron-lbaas port problem then we are good to go

drekle · 2019-05-29T13:44:03Z

@zetaab I have not changed ports to query all ports on the cluster network, and then delete them. Previously this was trying to grep by name. There was an empty name for dynamically created ports.

drekle · 2019-06-03T12:48:13Z

/hold cancel

zetaab · 2019-06-03T12:49:38Z

/approve
/lgtm

k8s-ci-robot · 2019-06-03T12:49:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: drekle, zetaab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/resources/openstack/OWNERS~~ [zetaab]
~~upup/pkg/fi/cloudup/openstack/OWNERS~~ [zetaab]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added area/provider/openstack Issues or PRs related to openstack provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 21, 2019

k8s-ci-robot requested review from rdrgmnzs and zetaab May 21, 2019 13:21

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label May 21, 2019

drekle commented May 21, 2019

View reviewed changes

drekle force-pushed the openstack_delete_dynamic_floating_ip branch from 36bd12b to a56b5c4 Compare May 21, 2019 13:33

zetaab requested changes May 21, 2019

View reviewed changes

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 22, 2019

Delete floating IP's based on the subnet resource, to clean up dynami…

7c53b35

…cally added ones as well, allowing for subnet deletion Bazel updates Updating deletion logic for octavia clusters

drekle force-pushed the openstack_delete_dynamic_floating_ip branch from a56b5c4 to 7c53b35 Compare May 23, 2019 13:53

zetaab approved these changes May 26, 2019

View reviewed changes

Openstack port resources derived from network

7b433cf

Removing Unused type

9542e9d

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 3, 2019

k8s-ci-robot assigned zetaab Jun 3, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 3, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 3, 2019

k8s-ci-robot merged commit d4bf54c into kubernetes:master Jun 3, 2019

This was referenced Jun 6, 2019

Release 1.13 backport 7045 #7106

Merged

Release 1 14 backport 7045 #7107

Merged

release 1.14 backport 7050 #7108

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Openstack delete dynamic floating ip in delete cluster #7045

Openstack delete dynamic floating ip in delete cluster #7045

drekle commented May 21, 2019

drekle May 21, 2019

zetaab May 21, 2019

drekle May 21, 2019 •

edited

Loading

drekle commented May 21, 2019

zetaab May 21, 2019

drekle commented May 22, 2019

drekle commented May 23, 2019

zetaab left a comment •

edited

Loading

drekle commented May 29, 2019

drekle commented Jun 3, 2019

zetaab commented Jun 3, 2019

k8s-ci-robot commented Jun 3, 2019

		var resourceTrackers []*resources.Resource

		if os.osCloud.UseOctavia() {

Openstack delete dynamic floating ip in delete cluster #7045

Openstack delete dynamic floating ip in delete cluster #7045

Conversation

drekle commented May 21, 2019

drekle May 21, 2019

Choose a reason for hiding this comment

zetaab May 21, 2019

Choose a reason for hiding this comment

drekle May 21, 2019 • edited Loading

Choose a reason for hiding this comment

drekle commented May 21, 2019

zetaab May 21, 2019

Choose a reason for hiding this comment

drekle commented May 22, 2019

drekle commented May 23, 2019

zetaab left a comment • edited Loading

Choose a reason for hiding this comment

drekle commented May 29, 2019

drekle commented Jun 3, 2019

zetaab commented Jun 3, 2019

k8s-ci-robot commented Jun 3, 2019

drekle May 21, 2019 •

edited

Loading

zetaab left a comment •

edited

Loading