Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Windows Networking Issues with ExpressRoute #1713

Closed
LiamLeane opened this issue Nov 5, 2017 · 5 comments
Closed

Windows Networking Issues with ExpressRoute #1713

LiamLeane opened this issue Nov 5, 2017 · 5 comments
Labels
orchestrator/k8s stale windows

Comments

@LiamLeane
Copy link

LiamLeane commented Nov 5, 2017 •
edited
Loading

Is this a request for help?:

Yes

Is this an ISSUE or FEATURE REQUEST? (choose one):

Issue

What version of acs-engine?:

0.8.0 & build from master (both have the same issue)

Orchestrator and version (e.g. Kubernetes, DC/OS, Swarm)

Kubernetes 1.6, 1.7 & 1.8 (all have the same issue)

What happened:
Windows based pods can reach other pods and services but cannot reach internet or internal (10.x) subnets which are peered in. Pods can reach other hosts on the same subnet (EG static IP of master).

Less important (easy to fix within the pod itself) Windows pods are not using kube-dns for resolving addresses, instead its pointing at the bridge gateway;

DNS Servers . . . . . . . . . . . : 172.29.176.1

Directly querying kube-dns works ok and resetting DNS on pod interfaces from within the pod works for correcting this issue;

PS C:\> nslookup www.google.com 192.168.64.2
Server:  kube-dns.kube-system.svc.cluster.local
Address:  192.168.64.2

Non-authoritative answer:
Name:    www.google.com
Addresses:  2607:f8b0:4006:812::2004
          172.217.10.132

What you expected to happen:
Windows pods to have network connectivity and set dns server correctly.

How to reproduce it (as minimally and precisely as possible):
Every possible configuration combination which uses a custom vnet peered to an ExpressRoute seems to cause this problem. As well as the configuration bellow I have tried different ip configurations, no direct ip configuration etc.

Anything else we need to know:
Linux based pods have both internet and access to peered subnets ok. Custom vnet on peered network is using 10.59.236.0/23 (all network interfaces attached to nodes are on this vnet and are reachable over the ExpressRoute), routing table is attached to the subnet. Same commands which fail on Windows based pod work on Windows node. 10.71.44.72 is an on-prem address on other side of ExpressRoute.

Linux based pod

/ # nslookup www.google.com
Server:    192.168.64.2
Address 1: 192.168.64.2 kube-dns.kube-system.svc.cluster.local

Name:      www.google.com
Address 1: 172.217.10.132 lga34s16-in-f4.1e100.net
Address 2: 2607:f8b0:4006:812::2004 lga34s16-in-x04.1e100.net
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=52 time=71.540 ms
64 bytes from 8.8.8.8: seq=1 ttl=52 time=70.774 ms
64 bytes from 8.8.8.8: seq=2 ttl=52 time=70.879 ms
64 bytes from 8.8.8.8: seq=3 ttl=52 time=70.593 ms
64 bytes from 8.8.8.8: seq=4 ttl=52 time=70.599 ms
64 bytes from 8.8.8.8: seq=5 ttl=52 time=70.690 ms
64 bytes from 8.8.8.8: seq=6 ttl=52 time=70.703 ms
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 7 packets received, 0% packet loss
round-trip min/avg/max = 70.593/70.825/71.540 ms
/ # ping 10.71.44.72
PING 10.71.44.72 (10.71.44.72): 56 data bytes
64 bytes from 10.71.44.72: seq=0 ttl=242 time=94.458 ms
64 bytes from 10.71.44.72: seq=1 ttl=242 time=94.226 ms
64 bytes from 10.71.44.72: seq=2 ttl=242 time=94.049 ms
64 bytes from 10.71.44.72: seq=3 ttl=242 time=94.149 ms
64 bytes from 10.71.44.72: seq=4 ttl=242 time=97.459 ms
--- 10.71.44.72 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 94.049/94.868/97.459 ms

Windows based pod;

PS C:\> nslookup www.google.com
Server:  UnKnown
Address:  172.29.176.1

*** UnKnown can't find www.google.com: Server failed
PS C:\> ping 8.8.8.8

Pinging 8.8.8.8 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 8.8.8.8:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),
PS C:\> ping 10.71.44.72

Pinging 10.71.44.72 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 10.71.44.72:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

acs-engine json;

{
  "apiVersion": "vlabs",
  "properties": {
    "orchestratorProfile": {
      "orchestratorType": "Kubernetes",
	  "orchestratorRelease": "1.7",
      "kubernetesConfig": {
		"clusterSubnet": "192.168.0.0/18",
		"serviceCidr": "192.168.64.0/18",
		"dnsServiceIP": "192.168.64.2",
		"nonMasqueradeCidr": "192.168.0.0/16"
      }
    },
    "masterProfile": {
      "count": 1,
      "dnsPrefix": "albuild",
      "vmSize": "Standard_DS2",
	  "vnetSubnetId": "/subscriptions/.../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork/subnets/Build",
      "firstConsecutiveStaticIP": "10.59.236.10"
    },
    "agentPoolProfiles": [
      {
        "name": "linux",
        "count": 1,
        "vmSize": "Standard_DS3",
        "availabilityProfile": "AvailabilitySet",		
        "OSDiskSizeGB": 200,
        "storageProfile" : "ManagedDisks",
	    "vnetSubnetId": "/subscriptions/.../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork/subnets/Build"
      },
      {
        "name": "windows",
        "count": 1,
        "vmSize": "Standard_DS4",
        "availabilityProfile": "AvailabilitySet",		
        "OSDiskSizeGB": 200,
        "storageProfile" : "ManagedDisks",
        "osType": "Windows",
		"vnetSubnetId": "/subscriptions/../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork/subnets/Build"
      }
    ],
    "windowsProfile": {
      "adminUsername": "...",
      "adminPassword": "..."
    },
    "linuxProfile": {
      "adminUsername": "aldevops",
      "ssh": {
        "publicKeys": [
          {
            "keyData":"..."
          }
        ]
      }
    },
    "servicePrincipalProfile": {
      "clientId": "...",
      "secret": "..."
    }
  }
}
@LiamLeane
Copy link
Author

Also confirming that switching the container network to NAT restores interwebs and gets traffic across the ExpressRoute, obviously not an ideal solution as this horribly breaks interpod communication. I can't see anything obviously misconfigured with the network itself;

[
    {
        "Name": "transparentNet",
        "Id": "fee7780573401e34a41d08807b459608846ed286458636f7b342bf6afdc05109",
        "Created": "2017-11-05T20:41:57.7674888Z",
        "Scope": "local",
        "Driver": "transparent",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "windows",
            "Options": null,
            "Config": [
                {
                    "Subnet": "192.168.2.0/24",
                    "Gateway": "192.168.2.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.windowsshim.hnsid": "40648ffb-1519-4cc5-ab6f-9aa9d88d40ca",
            "com.docker.network.windowsshim.networkname": "transparentNet"
        },
        "Labels": {}
    }
]

or the adapters themselves

Windows IP Configuration

   Host Name . . . . . . . . . . . . : 72151k8s9010
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : u0fjtlpro3ke5jajvx302uvw3e.bx.internal.cloudapp.net

Ethernet adapter vEthernet (HNS Internal NIC):

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter
   Physical Address. . . . . . . . . : 00-15-5D-12-FF-51
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::9c98:4cae:d9bd:7ded%12(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.29.176.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 218109277
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-90-A0-1F-00-15-5D-0A-1D-5C
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter vEthernet (HNSTransparent):

   Connection-specific DNS Suffix  . : u0fjtlpro3ke5jajvx302uvw3e.bx.internal.cloudapp.net
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #2
   Physical Address. . . . . . . . . : 00-0D-3A-19-B6-46
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::5dbf:a189:ebcb:ac7f%6(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.59.236.5(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.254.0
   Lease Obtained. . . . . . . . . . : Sunday, November 5, 2017 8:41:47 PM
   Lease Expires . . . . . . . . . . : Thursday, December 13, 2153 3:22:42 AM
   Default Gateway . . . . . . . . . : 10.59.236.1
   DHCP Server . . . . . . . . . . . : 168.63.129.16
   DHCPv6 IAID . . . . . . . . . . . : 251661626
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-90-A0-1F-00-15-5D-0A-1D-5C
   DNS Servers . . . . . . . . . . . : 168.63.129.16
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter vEthernet (forwarder):

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #3
   Physical Address. . . . . . . . . : 00-15-5D-0A-28-00
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::2996:9e9e:7911:acef%4(Preferred)
   IPv4 Address. . . . . . . . . . . : 192.168.2.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.64.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.64.2(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.72.97(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.83.157(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   IPv4 Address. . . . . . . . . . . : 192.168.91.76(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 369104221
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-90-A0-1F-00-15-5D-0A-1D-5C
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.u0fjtlpro3ke5jajvx302uvw3e.bx.internal.cloudapp.net:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{A92A8755-9ACC-46EF-A423-29753BB97BB2}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : u0fjtlpro3ke5jajvx302uvw3e.bx.internal.cloudapp.net
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{0467B881-63E2-4A29-B024-8223F5DA0ABE}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #3
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

@LiamLeane LiamLeane changed the title Windows Networking Issues Windows Networking Issues with ExpressRoute Nov 9, 2017
@jackfrancis
Copy link
Member

@JiangtianLi Is this a known behavioral limitation of Windows clusters w/ VNET?

@JiangtianLi
Copy link
Contributor

@jackfrancis This is possibly a limitation for RS1 Windows container networking.

@LiamLeane What is your apimodel and custom vnet setup config?

@madhanrm Is this a known issue to you?

@LiamLeane
Copy link
Author

LiamLeane commented Nov 14, 2017 •
edited
Loading

API model is at the end of the OP.

ARM for network is below. Fairly simple setup, single subnet 10.59.236.0/23 with an ExpressRoute peering. The only real curiosity with our setup beyond the ExpressRoute is keeping kubernetes networks outside of 10.0.0.0/8 as its used internally.

I confirmed this problem does not occur from a peered vnet and tried it in a different subscription with a peered network to confirm it wasn't subscription specific. Initially I wasn't sure if it was something up with routing or masquerading on the Windows node but adding routes to the Windows node to send internal (other end of the ExpressRoute) to the vnet gateway didn't resolve the issue.

Route table included at the end of this comment.

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "virtualNetworks_ClientVirtualizationNetwork_name": {
            "defaultValue": "ClientVirtualizationNetwork",
            "type": "String"
        },
        "subnets_Build_name": {
            "defaultValue": "Build",
            "type": "String"
        },
        "virtualNetworkPeerings_ClientVirtualizationNetwork_CTXITExpressRoute_name": {
            "defaultValue": "ClientVirtualizationNetwork-CTXITExpressRoute",
            "type": "String"
        },
        "virtualNetworks_ClientVirtualizationNetwork_id": {
            "defaultValue": "/subscriptions/.../resourceGroups/Build/providers/Microsoft.Network/routeTables/k8s-master-72151065-routetable",
            "type": "String"
        },
        "virtualNetworks_ClientVirtualizationNetwork_id_1": {
            "defaultValue": "/subscriptions/.../resourceGroups/expressroute-rg/providers/Microsoft.Network/virtualNetworks/CTXITExpressRoute",
            "type": "String"
        },
        "subnets_Build_id": {
            "defaultValue": "/subscriptions/.../resourceGroups/Build/providers/Microsoft.Network/routeTables/k8s-master-72151065-routetable",
            "type": "String"
        },
        "virtualNetworkPeerings_ClientVirtualizationNetwork_CTXITExpressRoute_id": {
            "defaultValue": "/subscriptions/.../resourceGroups/expressroute-rg/providers/Microsoft.Network/virtualNetworks/CTXITExpressRoute",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "comments": "Generalized from resource: '/subscriptions/.../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork'.",
            "type": "Microsoft.Network/virtualNetworks",
            "name": "[parameters('virtualNetworks_ClientVirtualizationNetwork_name')]",
            "apiVersion": "2017-06-01",
            "location": "eastus",
            "scale": null,
            "properties": {
                "provisioningState": "Succeeded",
                "resourceGuid": "ad998aa6-77f1-4f54-a409-adfbae52b6ec",
                "addressSpace": {
                    "addressPrefixes": [
                        "10.59.236.0/23"
                    ]
                },
                "dhcpOptions": {
                    "dnsServers": []
                },
                "subnets": [
                    {
                        "name": "Build",
                        "etag": "W/\"f1634d12-d1a0-4b69-8040-47efd3290f48\"",
                        "properties": {
                            "provisioningState": "Succeeded",
                            "addressPrefix": "10.59.236.0/23",
                            "routeTable": {
                                "id": "[parameters('virtualNetworks_ClientVirtualizationNetwork_id')]"
                            }
                        }
                    }
                ],
                "virtualNetworkPeerings": [
                    {
                        "name": "[concat(parameters('virtualNetworks_ClientVirtualizationNetwork_name'),'-CTXITExpressRoute')]",
                        "etag": "W/\"f1634d12-d1a0-4b69-8040-47efd3290f48\"",
                        "properties": {
                            "provisioningState": "Succeeded",
                            "peeringState": "Connected",
                            "remoteVirtualNetwork": {
                                "id": "[parameters('virtualNetworks_ClientVirtualizationNetwork_id_1')]"
                            },
                            "allowVirtualNetworkAccess": true,
                            "allowForwardedTraffic": false,
                            "allowGatewayTransit": false,
                            "useRemoteGateways": true
                        }
                    }
                ]
            },
            "dependsOn": []
        },
        {
            "comments": "Generalized from resource: '/subscriptions/.../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork/subnets/Build'.",
            "type": "Microsoft.Network/virtualNetworks/subnets",
            "name": "[concat(parameters('virtualNetworks_ClientVirtualizationNetwork_name'), '/', parameters('subnets_Build_name'))]",
            "apiVersion": "2017-06-01",
            "scale": null,
            "properties": {
                "provisioningState": "Succeeded",
                "addressPrefix": "10.59.236.0/23",
                "routeTable": {
                    "id": "[parameters('subnets_Build_id')]"
                }
            },
            "dependsOn": [
                "[resourceId('Microsoft.Network/virtualNetworks', parameters('virtualNetworks_ClientVirtualizationNetwork_name'))]"
            ]
        },
        {
            "comments": "Generalized from resource: '/subscriptions/.../resourceGroups/Networking/providers/Microsoft.Network/virtualNetworks/ClientVirtualizationNetwork/virtualNetworkPeerings/ClientVirtualizationNetwork-CTXITExpressRoute'.",
            "type": "Microsoft.Network/virtualNetworks/virtualNetworkPeerings",
            "name": "[concat(parameters('virtualNetworks_ClientVirtualizationNetwork_name'), '/', parameters('virtualNetworkPeerings_ClientVirtualizationNetwork_CTXITExpressRoute_name'))]",
            "apiVersion": "2017-06-01",
            "scale": null,
            "properties": {
                "provisioningState": "Succeeded",
                "peeringState": "Connected",
                "remoteVirtualNetwork": {
                    "id": "[parameters('virtualNetworkPeerings_ClientVirtualizationNetwork_CTXITExpressRoute_id')]"
                },
                "allowVirtualNetworkAccess": true,
                "allowForwardedTraffic": false,
                "allowGatewayTransit": false,
                "useRemoteGateways": true
            },
            "dependsOn": [
                "[resourceId('Microsoft.Network/virtualNetworks', parameters('virtualNetworks_ClientVirtualizationNetwork_name'))]"
            ]
        }
    ]
}

===========================================================================
Interface List
 12...00 15 5d 12 ff 51 ......Hyper-V Virtual Ethernet Adapter
  6...00 0d 3a 19 b6 46 ......Hyper-V Virtual Ethernet Adapter #2
  4...00 15 5d 0a 28 00 ......Hyper-V Virtual Ethernet Adapter #3
  1...........................Software Loopback Interface 1
  3...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
  2...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 10...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #3
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0      10.59.236.1      10.59.236.5     15
      10.59.236.0    255.255.254.0         On-link       10.59.236.5    271
      10.59.236.5  255.255.255.255         On-link       10.59.236.5    271
    10.59.237.255  255.255.255.255         On-link       10.59.236.5    271
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
    168.63.129.16  255.255.255.255      10.59.236.1      10.59.236.5     16
  169.254.169.254  255.255.255.255      10.59.236.1      10.59.236.5     16
     172.29.176.0    255.255.240.0         On-link      172.29.176.1    271
     172.29.176.1  255.255.255.255         On-link      172.29.176.1    271
   172.29.191.255  255.255.255.255         On-link      172.29.176.1    271
      192.168.2.0    255.255.255.0         On-link       192.168.2.1    271
      192.168.2.1  255.255.255.255         On-link       192.168.2.1    271
    192.168.2.255  255.255.255.255         On-link       192.168.2.1    271
     192.168.64.0    255.255.255.0         On-link       192.168.2.1    271
     192.168.64.1  255.255.255.255         On-link       192.168.2.1    271
     192.168.64.2  255.255.255.255         On-link       192.168.2.1    271
   192.168.64.255  255.255.255.255         On-link       192.168.2.1    271
     192.168.69.0    255.255.255.0         On-link       192.168.2.1    271
    192.168.69.55  255.255.255.255         On-link       192.168.2.1    271
   192.168.69.255  255.255.255.255         On-link       192.168.2.1    271
     192.168.72.0    255.255.255.0         On-link       192.168.2.1    271
    192.168.72.97  255.255.255.255         On-link       192.168.2.1    271
   192.168.72.255  255.255.255.255         On-link       192.168.2.1    271
     192.168.83.0    255.255.255.0         On-link       192.168.2.1    271
   192.168.83.157  255.255.255.255         On-link       192.168.2.1    271
   192.168.83.255  255.255.255.255         On-link       192.168.2.1    271
     192.168.91.0    255.255.255.0         On-link       192.168.2.1    271
    192.168.91.76  255.255.255.255         On-link       192.168.2.1    271
   192.168.91.255  255.255.255.255         On-link       192.168.2.1    271
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link      172.29.176.1    271
        224.0.0.0        240.0.0.0         On-link       10.59.236.5    271
        224.0.0.0        240.0.0.0         On-link       192.168.2.1    271
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link      172.29.176.1    271
  255.255.255.255  255.255.255.255         On-link       10.59.236.5    271
  255.255.255.255  255.255.255.255         On-link       192.168.2.1    271
===========================================================================
Persistent Routes:
  None

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    331 ::1/128                  On-link
  1    331 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

@stale
Copy link

stale bot commented Mar 9, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contribution. Note that acs-engine is deprecated--see https://github.com/Azure/aks-engine instead.

@stale stale bot added the stale label Mar 9, 2019
@stale stale bot closed this as completed Mar 16, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
orchestrator/k8s stale windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@LiamLeane @jackfrancis @JiangtianLi