How to set up Container to Internal (Peered) Network

[dsemarketer]

Hi, I have an Azure ACS Kubernetes cluster running. Its a custom vnet which I have working correctly.

I have older legacy applications running in the same Azure subscription on VMs (different vnet) with connectivity through Peering. I can ping the cluster from the legacy machines and communicate with pods if I expose them on via NodePort. I can also ping the legacy machines from the cluster nodes and the master.

The pods / containers running inside the cluster can access each other properly and access the internet if necessary. They can not access the peered network / legacy apps. I suspect its either a route i need to add into the docker / kubernetes network or iptables / firewall stopping the communication

How can I get my containers to talk to the peered subnet?

For reference:
Kubernetes cluster 10.10.x.x/16 (service network 10.10.0.0/20)
Agents 10.10.16.x/20
Container internal network (auto created - the ips the pods get from the host) 10.240.1.x/24
Peered network 10.12.x.x/16

A pod (10.240.1.5) can communicate with the internet, the master ip, agent (host) ips and other containers

But it can't connect to 10.12.x.x/16 the peered network.

[colemickens]

Can you try to mirror the routeTable in the Kubernetes cluster into the routeTable on the peered vnet (will need to try it manually and/or script it for a longer term solution)? The last time someone reported this issue, this had apparently solved their problem.

Alternatively, I think if the ip masquerading were adjusted, I think traffic to 10.12.x.x/16 could be masqeuraded as the node, and thus not require the peer route table to be updated, but this is getting to the limits of my iptables knowledge. @anhowe or @brendandburns might know more about if masquerading 10.12.x.x./16 traffic the same as we masquerade internet traffic would suffice.

Regarding masquerading... this issue is relevant: kubernetes/kubernetes#11204

[dsemarketer]

Fixed. Thank you so much!

It was a masquerading issue. What I had to do was add the rule to the linux agent:
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE -d 10.12.x.x/16

[colemickens]

Very cool. Thank you for confirming, I can now pass along this info with more confidence in the future!

[siamak-haschemi]

I was needing that two weeks ago :-) Took me two days to figure out. Thank you @colemickens for all of your support here.

EDIT: I would like to extend the question. How do we persist the iptables rules? Can we put them somehow into the ARM templates? Can we use the acs-engine for that?

[colemickens]

It's a miracle I saw your edit. Please don't do that. I don't re-check these threads unless I get an email.

Yes, you could add the iptables rule to ACS-Engine in customData to make it "persist".

(Alternatively, if you can make the peer vnet outside of 10.x, it should masquerade traffic by default. Not sure if that's an option for you...)

[siamak-haschemi]

Sorry, will never do that again ...

I will try our your suggestion with the customData and report back.

[dsemarketer]

To save them as persistent, I used iptables-save > /etc/iptables/rules.v4 to get the rule needed and then installed iptables-persistent to auto load it. I then rebooted one of the nodes to check and the rule was reloaded automatically.

[himikof]

This could be solved by passing --non-masquerade-cidr=10.240.1.0/24 (pod network address range) to kubelet on all nodes. This replaces the default value of 10.0.0.0/8 and fixes MASQUERADE iptables rule in cases when some external vnet is in the 10.0.0.0/8 address range.

[andrewstevenson]

@himikof Hi, I'm trying to figure out where I need to set this? (kubernetes newbie)

[himikof]

Kubelet arguments are set in /etc/systemd/system/kubelet.service on all nodes after deployment, or in parts/kuberneteskubelet.service in acs-engine templates.

[sylr]

I'm wondering if this could not be generated by acs-engine ?

[sylr]

Something like that sylr@73ba26f

Does someone can tell me what is required to do when changing files in parts/ ? After this patch acs-engine generate outputs does not apply the changes I made to parts/ .

[JackQuincy]

You need to run go generate. Make build should do that though... If not I'd consider it a big

[sylr]

PS C:\gopath\src\github.com\Azure\acs-engine> git st
On branch non-masquerade-cidr
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   parts/kubernetesagentcustomdata.yml
        modified:   parts/kuberneteskubelet.service

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        examples/k8sx-EUW-dev.json
        pwet.patch

PS C:\gopath\src\github.com\Azure\acs-engine> go generate
PS C:\gopath\src\github.com\Azure\acs-engine>
PS C:\gopath\src\github.com\Azure\acs-engine> git st
On branch non-masquerade-cidr
Changes to be committed:
  (use "git reset HEAD <file>..." to unstage)

        modified:   parts/kubernetesagentcustomdata.yml
        modified:   parts/kuberneteskubelet.service

Untracked files:
  (use "git add <file>..." to include in what will be committed)

        examples/k8sx-EUW-dev.json
        pwet.patch

I've tried go generate but it does nothing.

[sylr]

All right, I made it but guys, you should really update the build docs and mention that you need to add $GOPATH/bin to $PATH

[jackfrancis]

I'm not sure there's anything actionable here, sorry for the devx inconvenience over the summer @sylr