ACI pod cannot communicate with other pods

[vedant-khandelwal]

What happened:

Deployed an AKS cluster with virtual node enabled. The pods running in ACI get IP addresses not in the same range as AKS vnet. The ACI pod has a corresponding service and other pods also have corresponding services. As a result those pods are not able to communicate with other pods running in AKS.

What you expected to happen:

ACI pod should be able to communicate with other pods using service names.

How to reproduce it (as minimally and precisely as possible):

Just create an AKS cluster with virtual node enabled and run pods with services in both AKS nodes and ACI.

Environment:

• Kubernetes version : 1.13.5
• Size of cluster : 3 nodes
• General description of workloads in the cluster: HTTP microservices

[vedant-khandelwal]

I find that the virtual kubelet pod doesn't have ACI subnet defined, but when I try to edit the deployment, it gets reset and becomes empty so I can't set the subnet. Not sure how it got reset

[vedant-khandelwal]

disabled and re-enabled the virtual node feature in the cluster and set the subnet again. But the pod is still unable to discover services running on other nodes in the cluster.

[srrengar]

The ACI containers that are created on the virtual node are in their own delegated subnet but all within the same VNet as the AKS cluster. Are you creating the delegated subnet yourself or letting the portal do it for you? What errors are you seeing?

[vedant-khandelwal]

Azure created the subnet for me. I just enabled virtual node option on the portal. I'm seeing dns resolution failures for FQDN service names in pods running in ACI

…

On Tue, May 14, 2019 at 11:57 PM Sravan Rengarajan ***@***.***> wrote: The ACI containers that are created on the virtual node are in their own delegated subnet but all within the same VNet as the AKS cluster. Are you creating the delegated subnet yourself or letting the portal do it for you? What errors are you seeing? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#868?email_source=notifications&email_token=AHPR3O6NFPSNXPOEIGMX4NLPVMABJA5CNFSM4G6OIHX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVMLW2I#issuecomment-492354409>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHPR3O5JLILPBVPOQMMNENDPVMABJANCNFSM4G6OIHXQ> .

[srrengar]

@robbiezhang any reason why this would happen?

[robbiezhang]

@vedant-khandelwal are you using FQDN ([service name].[namespace].svc.cluster.local) or the short form ([service name].[namespace])? Can you resolve public DNS?
Similar issue: virtual-kubelet/virtual-kubelet#456

[vedant-khandelwal]

I'm using the FQDN. Could this issue be because we've configured the kubernetes load balancer to route external requests internally

…

On Fri, 17 May, 2019, 11:24 PM Robbie Zhang, ***@***.***> wrote: @vedant-khandelwal <https://github.com/vedant-khandelwal> are you using FQDN ([service name].[namespace].svc.cluster.local) or the short form ([service name].[namespace])? Can you resolve public DNS? Similar issue: virtual-kubelet/virtual-kubelet#456 <virtual-kubelet/virtual-kubelet#456> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#868?email_source=notifications&email_token=AHPR3O3DBYSZ7NR6LUCKH5TPV3WOHA5CNFSM4G6OIHX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVVN2SY#issuecomment-493542731>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHPR3O42GSICXXYC3DWXM53PV3WOHANCNFSM4G6OIHXQ> .

[srrengar]

@vedant-khandelwal is this still occurring?

Tagging @ibabou

[vedant-khandelwal]

Yes this is still an issue. Can any of you please schedule a call to discuss this issue? We have been unable to use ACI due to these issues.

…

On Tue, Aug 27, 2019 at 1:59 AM Sravan Rengarajan ***@***.***> wrote: @vedant-khandelwal <https://github.com/vedant-khandelwal> is this still occurring? Tagging @ibabou <https://github.com/ibabou> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#868?email_source=notifications&email_token=AHPR3O73PDNOUGKAJDE673LQGQ4K3A5CNFSM4G6OIHX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5FSQHA#issuecomment-525019164>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHPR3O7HIFQEYOCPKMNLAJTQGQ4K3ANCNFSM4G6OIHXQ> .

[srrengar]

Have you opened a support case on this?

[vedant-khandelwal]

Yes I have. I have also provided all the information they asked for. Haven't heard back from them for quite a while now.

…

On Tue, Aug 27, 2019 at 10:28 PM Sravan Rengarajan ***@***.***> wrote: Have you opened a support case on this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#868?email_source=notifications&email_token=AHPR3OZMMCC7YEZYBUU2MO3QGVMKVA5CNFSM4G6OIHX2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5INKTA#issuecomment-525391180>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHPR3O4A2ZH4KLEZNX4F6ITQGVMKVANCNFSM4G6OIHXQ> .

[srrengar]

What is the support case #?

[jluk]

@vedant-khandelwal I am closing this issue due to 3 weeks since last ask. Please feel free to open another support ticket for this virtual nodes issue, which is better handled through support ticket than github issue anyways. Thanks!