Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[occm] 'internal-network-name' broken since release 1.18 #1039

Closed
gdetal opened this issue Apr 20, 2020 · 4 comments · Fixed by #1041
Closed

[occm] 'internal-network-name' broken since release 1.18 #1039

gdetal opened this issue Apr 20, 2020 · 4 comments · Fixed by #1041
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@gdetal
Copy link

gdetal commented Apr 20, 2020

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

I configured internal-network-name in the cloud.conf configuration file. But the InternalIP of my minion nodes always get the wrong IP address.

What you expected to happen:

That the InternalIP of my minion nodes corresponds to the one allocated in the network specified by internal-network-name.

How to reproduce it:

Create a cluster with minion that have more than one network. Specify one of the networks under internal-network-name and create minions. Depending on the ordering of the VM's ports the wrong node InternalIP will be patched.

Anything else we need to know?:

I've tested releases 1.17, 1.18 and master. Only the latter two are broken. After increasing log verbosity to 5 for occm I can see the following for those releases:

I0420 07:59:30.338203       1 leaderelection.go:272] successfully renewed lease kube-system/cloud-controller-manager
I0420 07:59:30.443443       1 openstack.go:754] Node 'openstack-md-0-x92kl' address '10.8.0.54' ignored due to 'internal-network-name' option
I0420 07:59:30.443465       1 openstack.go:754] Node 'openstack-md-0-x92kl' address '10.7.0.246' ignored due to 'internal-network-name' option
I0420 07:59:30.443471       1 openstack.go:754] Node 'openstack-md-0-x92kl' address '10.9.0.110' ignored due to 'internal-network-name' option
I0420 07:59:30.443477       1 openstack_instances.go:90] NodeAddresses(openstack-md-0-x92kl) => [{InternalIP 10.7.0.246} {InternalIP 10.9.0.110} {InternalIP 10.8.0.54} {InternalIP 10.6.0.202}]

It seems that commit 46cab81 broke #444 as the first loop do not use the internal-network-name option.

Environment:

  • openstack-cloud-controller-manager(or other related binary) version: 1.18 // master
  • OpenStack version:
  • Others:
@k8s-ci-robot k8s-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 20, 2020
@kayrus
Copy link
Contributor

kayrus commented Apr 20, 2020 •
edited
Loading

@gdetal can you set verbose debugging (--v=6) and provide the internal-network-name value? Using --v=6 can show the actual server detailed JSON object returned by OpenStack Nova, e.g.

        "addresses": {
            "private": [
                {
                    "OS-EXT-IPS-MAC:mac_addr": "00:0c:29:0d:11:74",
                    "OS-EXT-IPS:type": "fixed",
                    "addr": "192.168.1.30",
                    "version": 4
                }
            ]
        },

private key should correspond to internal-network-name value.

@gdetal
Copy link
Author

gdetal commented Apr 20, 2020

thanks for the fast follow-up.

Find below the requested output with --v=6:

I0420 09:25:52.577018       1 openstack.go:653] OpenStack Request URL: GET http://***/v2.1/2867c293ed9a450b9b214e08060c962b/servers/detail?name=%5Eopenstack-md-0-wmt58%24
I0420 09:25:52.577054       1 openstack.go:653] OpenStack Request Headers:
I0420 09:25:52.577062       1 openstack.go:653] Accept: application/json
I0420 09:25:52.577069       1 openstack.go:653] User-Agent: openstack-cloud-controller-manager/latest gophercloud/2.0.0
I0420 09:25:52.577075       1 openstack.go:653] X-Auth-Token: ***
I0420 09:25:52.897774       1 openstack.go:653] OpenStack Response Code: 200
I0420 09:25:52.897876       1 openstack.go:653] OpenStack Response Headers:
I0420 09:25:52.897889       1 openstack.go:653] Content-Length: 2294
I0420 09:25:52.897896       1 openstack.go:653] Content-Type: application/json
I0420 09:25:52.897902       1 openstack.go:653] Date: Mon, 20 Apr 2020 09:25:52 GMT
I0420 09:25:52.897908       1 openstack.go:653] Openstack-Api-Version: compute 2.1
I0420 09:25:52.897914       1 openstack.go:653] Vary: OpenStack-API-Version X-OpenStack-Nova-API-Version
I0420 09:25:52.897919       1 openstack.go:653] X-Compute-Request-Id: req-eddf8206-5c65-4492-acab-144ff9f2bd47
I0420 09:25:52.897925       1 openstack.go:653] X-Openstack-Nova-Api-Version: 2.1
I0420 09:25:52.897931       1 openstack.go:653] X-Openstack-Request-Id: req-eddf8206-5c65-4492-acab-144ff9f2bd47
I0420 09:25:52.898393       1 openstack.go:653] OpenStack Response Body: {
I0420 09:25:52.898408       1 openstack.go:653]   "servers": [
I0420 09:25:52.898415       1 openstack.go:653]     {
I0420 09:25:52.898424       1 openstack.go:653]       "OS-DCF:diskConfig": "MANUAL",
I0420 09:25:52.898431       1 openstack.go:653]       "OS-EXT-AZ:availability_zone": "nova",
I0420 09:25:52.898437       1 openstack.go:653]       "OS-EXT-SRV-ATTR:host": "***",
I0420 09:25:52.898443       1 openstack.go:653]       "OS-EXT-SRV-ATTR:hypervisor_hostname": "***",
I0420 09:25:52.898450       1 openstack.go:653]       "OS-EXT-SRV-ATTR:instance_name": "instance-000032bf",
I0420 09:25:52.898457       1 openstack.go:653]       "OS-EXT-STS:power_state": 1,
I0420 09:25:52.898468       1 openstack.go:653]       "OS-EXT-STS:task_state": null,
I0420 09:25:52.898475       1 openstack.go:653]       "OS-EXT-STS:vm_state": "active",
I0420 09:25:52.898483       1 openstack.go:653]       "OS-SRV-USG:launched_at": "2020-04-20T09:25:32.000000",
I0420 09:25:52.898494       1 openstack.go:653]       "OS-SRV-USG:terminated_at": null,
I0420 09:25:52.898501       1 openstack.go:653]       "accessIPv4": "",
I0420 09:25:52.898507       1 openstack.go:653]       "accessIPv6": "",
I0420 09:25:52.898514       1 openstack.go:653]       "addresses": {
I0420 09:25:52.898520       1 openstack.go:653]         "capi-int-net": [
I0420 09:25:52.898526       1 openstack.go:653]           {
I0420 09:25:52.898533       1 openstack.go:653]             "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:d8:30:ad",
I0420 09:25:52.898545       1 openstack.go:653]             "OS-EXT-IPS:type": "fixed",
I0420 09:25:52.898551       1 openstack.go:653]             "addr": "10.8.0.228",
I0420 09:25:52.898559       1 openstack.go:653]             "version": 4
I0420 09:25:52.898570       1 openstack.go:653]           }
I0420 09:25:52.898579       1 openstack.go:653]         ],
I0420 09:25:52.898585       1 openstack.go:653]         "capi-main-net": [
I0420 09:25:52.898592       1 openstack.go:653]           {
I0420 09:25:52.898598       1 openstack.go:653]             "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:20:ef:1f",
I0420 09:25:52.898604       1 openstack.go:653]             "OS-EXT-IPS:type": "fixed",
I0420 09:25:52.898611       1 openstack.go:653]             "addr": "10.7.0.169",
I0420 09:25:52.898619       1 openstack.go:653]             "version": 4
I0420 09:25:52.898631       1 openstack.go:653]           }
I0420 09:25:52.898638       1 openstack.go:653]         ],
I0420 09:25:52.898645       1 openstack.go:653]         "capi-slave-net": [
I0420 09:25:52.898656       1 openstack.go:653]           {
I0420 09:25:52.898662       1 openstack.go:653]             "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:8e:78:2c",
I0420 09:25:52.898669       1 openstack.go:653]             "OS-EXT-IPS:type": "fixed",
I0420 09:25:52.898675       1 openstack.go:653]             "addr": "10.9.0.180",
I0420 09:25:52.898682       1 openstack.go:653]             "version": 4
I0420 09:25:52.898688       1 openstack.go:653]           }
I0420 09:25:52.898694       1 openstack.go:653]         ],
I0420 09:25:52.898702       1 openstack.go:653]         "k8s-clusterapi-cluster-default-openstack": [
I0420 09:25:52.898712       1 openstack.go:653]           {
I0420 09:25:52.898723       1 openstack.go:653]             "OS-EXT-IPS-MAC:mac_addr": "fa:16:3e:01:f3:e2",
I0420 09:25:52.898729       1 openstack.go:653]             "OS-EXT-IPS:type": "fixed",
I0420 09:25:52.898735       1 openstack.go:653]             "addr": "10.6.0.236",
I0420 09:25:52.898742       1 openstack.go:653]             "version": 4
I0420 09:25:52.898749       1 openstack.go:653]           }
I0420 09:25:52.898755       1 openstack.go:653]         ]
I0420 09:25:52.898761       1 openstack.go:653]       },
I0420 09:25:52.898767       1 openstack.go:653]       "config_drive": "",
I0420 09:25:52.898774       1 openstack.go:653]       "created": "2020-04-20T09:24:40Z",
I0420 09:25:52.898780       1 openstack.go:653]       "flavor": {
I0420 09:25:52.898787       1 openstack.go:653]         "id": "fff52fd6-b47c-47fe-8b39-8b6f16465c19",
I0420 09:25:52.898798       1 openstack.go:653]         "links": [
I0420 09:25:52.898830       1 openstack.go:653]           {
I0420 09:25:52.898842       1 openstack.go:653]             "href": "http://***/2867c293ed9a450b9b214e08060c962b/flavors/fff52fd6-b47c-47fe-8b39-8b6f16465c19",
I0420 09:25:52.898849       1 openstack.go:653]             "rel": "bookmark"
I0420 09:25:52.898855       1 openstack.go:653]           }
I0420 09:25:52.898861       1 openstack.go:653]         ]
I0420 09:25:52.898868       1 openstack.go:653]       },
I0420 09:25:52.898875       1 openstack.go:653]       "hostId": "21ca266b5ebd8cda22aac315cb914d13c5afe01a1f4846e6f04c667b",
I0420 09:25:52.898885       1 openstack.go:653]       "id": "9396725d-ccf9-420c-9950-907ea86d000e",
I0420 09:25:52.898894       1 openstack.go:653]       "image": {
I0420 09:25:52.898901       1 openstack.go:653]         "id": "23295685-4cc3-4814-acc7-d707487c84e2",
I0420 09:25:52.898907       1 openstack.go:653]         "links": [
I0420 09:25:52.898913       1 openstack.go:653]           {
I0420 09:25:52.898920       1 openstack.go:653]             "href": "http://***/2867c293ed9a450b9b214e08060c962b/images/23295685-4cc3-4814-acc7-d707487c84e2",
I0420 09:25:52.898926       1 openstack.go:653]             "rel": "bookmark"
I0420 09:25:52.898933       1 openstack.go:653]           }
I0420 09:25:52.898939       1 openstack.go:653]         ]
I0420 09:25:52.898945       1 openstack.go:653]       },
I0420 09:25:52.898953       1 openstack.go:653]       "key_name": null,
I0420 09:25:52.898960       1 openstack.go:653]       "links": [
I0420 09:25:52.898971       1 openstack.go:653]         {
I0420 09:25:52.898977       1 openstack.go:653]           "href": "http://***/v2.1/2867c293ed9a450b9b214e08060c962b/servers/9396725d-ccf9-420c-9950-907ea86d000e",
I0420 09:25:52.898984       1 openstack.go:653]           "rel": "self"
I0420 09:25:52.898995       1 openstack.go:653]         },
I0420 09:25:52.899002       1 openstack.go:653]         {
I0420 09:25:52.899008       1 openstack.go:653]           "href": "http://***/2867c293ed9a450b9b214e08060c962b/servers/9396725d-ccf9-420c-9950-907ea86d000e",
I0420 09:25:52.899014       1 openstack.go:653]           "rel": "bookmark"
I0420 09:25:52.899021       1 openstack.go:653]         }
I0420 09:25:52.899030       1 openstack.go:653]       ],
I0420 09:25:52.899036       1 openstack.go:653]       "metadata": {},
I0420 09:25:52.899048       1 openstack.go:653]       "name": "openstack-md-0-wmt58",
I0420 09:25:52.899054       1 openstack.go:653]       "os-extended-volumes:volumes_attached": [],
I0420 09:25:52.899061       1 openstack.go:653]       "progress": 0,
I0420 09:25:52.899072       1 openstack.go:653]       "status": "ACTIVE",
I0420 09:25:52.899078       1 openstack.go:653]       "tenant_id": "2867c293ed9a450b9b214e08060c962b",
I0420 09:25:52.899084       1 openstack.go:653]       "updated": "2020-04-20T09:25:32Z",
I0420 09:25:52.899091       1 openstack.go:653]       "user_id": "f39435ff4ec527a5b2ad776e45592f943ca8f118ff96009c1af51b66eb541cec"
I0420 09:25:52.899098       1 openstack.go:653]     }
I0420 09:25:52.899104       1 openstack.go:653]   ]
I0420 09:25:52.899110       1 openstack.go:653] }
I0420 09:25:52.900126       1 openstack.go:822] OpenStack Request URL: GET http://***/v2.1/2867c293ed9a450b9b214e08060c962b/servers/9396725d-ccf9-420c-9950-907ea86d000e/os-interface
I0420 09:25:52.900164       1 openstack.go:822] OpenStack Request Headers:
I0420 09:25:52.900174       1 openstack.go:822] Accept: application/json
I0420 09:25:52.900181       1 openstack.go:822] User-Agent: openstack-cloud-controller-manager/latest gophercloud/2.0.0
I0420 09:25:52.900188       1 openstack.go:822] X-Auth-Token: ***
I0420 09:25:53.056975       1 openstack.go:822] OpenStack Response Code: 200
I0420 09:25:53.057034       1 openstack.go:822] OpenStack Response Headers:
I0420 09:25:53.057041       1 openstack.go:822] Content-Length: 1058
I0420 09:25:53.057048       1 openstack.go:822] Content-Type: application/json
I0420 09:25:53.057054       1 openstack.go:822] Date: Mon, 20 Apr 2020 09:25:52 GMT
I0420 09:25:53.057068       1 openstack.go:822] Openstack-Api-Version: compute 2.1
I0420 09:25:53.057074       1 openstack.go:822] Vary: OpenStack-API-Version X-OpenStack-Nova-API-Version
I0420 09:25:53.057098       1 openstack.go:822] X-Compute-Request-Id: req-76fbb6ef-08fa-448a-adaa-899c7c5ae156
I0420 09:25:53.057117       1 openstack.go:822] X-Openstack-Nova-Api-Version: 2.1
I0420 09:25:53.057124       1 openstack.go:822] X-Openstack-Request-Id: req-76fbb6ef-08fa-448a-adaa-899c7c5ae156
I0420 09:25:53.057887       1 openstack.go:822] OpenStack Response Body: {
I0420 09:25:53.057906       1 openstack.go:822]   "interfaceAttachments": [
I0420 09:25:53.057914       1 openstack.go:822]     {
I0420 09:25:53.057922       1 openstack.go:822]       "fixed_ips": [
I0420 09:25:53.057934       1 openstack.go:822]         {
I0420 09:25:53.057941       1 openstack.go:822]           "ip_address": "10.9.0.180",
I0420 09:25:53.057948       1 openstack.go:822]           "subnet_id": "739ddf8c-23f4-478d-9093-d9de87957519"
I0420 09:25:53.057956       1 openstack.go:822]         }
I0420 09:25:53.057963       1 openstack.go:822]       ],
I0420 09:25:53.057975       1 openstack.go:822]       "mac_addr": "fa:16:3e:8e:78:2c",
I0420 09:25:53.057981       1 openstack.go:822]       "net_id": "ac50b2eb-bc4f-4cc1-84ef-220532fec832",
I0420 09:25:53.057988       1 openstack.go:822]       "port_id": "02fc2748-3abf-4f02-9dbc-d1e9d4240969",
I0420 09:25:53.057995       1 openstack.go:822]       "port_state": "ACTIVE"
I0420 09:25:53.058002       1 openstack.go:822]     },
I0420 09:25:53.058012       1 openstack.go:822]     {
I0420 09:25:53.058024       1 openstack.go:822]       "fixed_ips": [
I0420 09:25:53.058031       1 openstack.go:822]         {
I0420 09:25:53.058038       1 openstack.go:822]           "ip_address": "10.6.0.236",
I0420 09:25:53.058044       1 openstack.go:822]           "subnet_id": "5be8e33b-b2d8-4631-bf85-1aa0db55b2ce"
I0420 09:25:53.058054       1 openstack.go:822]         }
I0420 09:25:53.058061       1 openstack.go:822]       ],
I0420 09:25:53.058067       1 openstack.go:822]       "mac_addr": "fa:16:3e:01:f3:e2",
I0420 09:25:53.058074       1 openstack.go:822]       "net_id": "32ff6843-e33f-455d-9c22-16e650594ca9",
I0420 09:25:53.058094       1 openstack.go:822]       "port_id": "38cbb77b-d6a7-4948-93d3-440d35f40e3d",
I0420 09:25:53.058102       1 openstack.go:822]       "port_state": "ACTIVE"
I0420 09:25:53.058109       1 openstack.go:822]     },
I0420 09:25:53.058116       1 openstack.go:822]     {
I0420 09:25:53.058123       1 openstack.go:822]       "fixed_ips": [
I0420 09:25:53.058133       1 openstack.go:822]         {
I0420 09:25:53.058140       1 openstack.go:822]           "ip_address": "10.8.0.228",
I0420 09:25:53.058151       1 openstack.go:822]           "subnet_id": "f538a607-9a5c-4b2f-a087-eaddcbc454ba"
I0420 09:25:53.058158       1 openstack.go:822]         }
I0420 09:25:53.058164       1 openstack.go:822]       ],
I0420 09:25:53.058171       1 openstack.go:822]       "mac_addr": "fa:16:3e:d8:30:ad",
I0420 09:25:53.058179       1 openstack.go:822]       "net_id": "4ef40a7f-040d-41c6-858c-35ac1c533ede",
I0420 09:25:53.058186       1 openstack.go:822]       "port_id": "725bb725-9c7c-4738-9dca-6d628338c4fe",
I0420 09:25:53.058193       1 openstack.go:822]       "port_state": "ACTIVE"
I0420 09:25:53.058199       1 openstack.go:822]     },
I0420 09:25:53.058209       1 openstack.go:822]     {
I0420 09:25:53.058216       1 openstack.go:822]       "fixed_ips": [
I0420 09:25:53.058222       1 openstack.go:822]         {
I0420 09:25:53.058228       1 openstack.go:822]           "ip_address": "10.7.0.169",
I0420 09:25:53.058235       1 openstack.go:822]           "subnet_id": "4a7be82e-34c0-46c5-9bea-267821aeb8cc"
I0420 09:25:53.058241       1 openstack.go:822]         }
I0420 09:25:53.058248       1 openstack.go:822]       ],
I0420 09:25:53.058255       1 openstack.go:822]       "mac_addr": "fa:16:3e:20:ef:1f",
I0420 09:25:53.058262       1 openstack.go:822]       "net_id": "f9e9abfe-5c7c-4fb5-9360-5a006136aac2",
I0420 09:25:53.058276       1 openstack.go:822]       "port_id": "f0fbdd87-15ba-4425-a936-a730ef25a743",
I0420 09:25:53.058283       1 openstack.go:822]       "port_state": "ACTIVE"
I0420 09:25:53.058294       1 openstack.go:822]     }
I0420 09:25:53.058301       1 openstack.go:822]   ]
I0420 09:25:53.058307       1 openstack.go:822] }
I0420 09:25:53.058867       1 openstack.go:754] Node 'openstack-md-0-wmt58' address '10.8.0.228' ignored due to 'internal-network-name' option
I0420 09:25:53.058880       1 openstack.go:754] Node 'openstack-md-0-wmt58' address '10.7.0.169' ignored due to 'internal-network-name' option
I0420 09:25:53.058885       1 openstack.go:754] Node 'openstack-md-0-wmt58' address '10.9.0.180' ignored due to 'internal-network-name' option
I0420 09:25:53.058899       1 openstack_instances.go:90] NodeAddresses(openstack-md-0-wmt58) => [{InternalIP 10.9.0.180} {InternalIP 10.6.0.236} {InternalIP 10.8.0.228} {InternalIP 10.7.0.169}]

Note here is also the relevant cloud.conf section:

[Networking]
public-network-name=public-net
internal-network-name=k8s-clusterapi-cluster-default-openstack

@kayrus
Copy link
Contributor

kayrus commented Apr 20, 2020

@gdetal looks like #884 logic missed networkingOpts.InternalNetworkName condition for interfaces. I'll try to fix it.

@gdetal
Copy link
Author

gdetal commented Apr 20, 2020

@kayrus ok, thank you !

@kayrus kayrus mentioned this issue Apr 20, 2020
Merged
@zuzzas zuzzas mentioned this issue Apr 23, 2020
Closed
@lingxiankong lingxiankong mentioned this issue May 3, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
3 participants
@gdetal @kayrus @k8s-ci-robot