Support hostPort with CNI

[cknowles]

According to kubernetes/kubernetes#23920 (comment) we will need to update versions of Calico and configuration to enable this on k8s 1.7. I'm not sure what if anything needs to change for flannel to also work.

[mumoshu]

@c-knowles Thanks for the info!
Perhaps I'll verify it to work until v0.9.8-rc.1 and then remove the note in the limitations doc.

[cknowles]

Sure, let's verify that I can try on our clusters. FYI I hit this from datadog's latest install notes which use the host port functionality.

[andrejvanderzee]

Ha! This just cost me a couple of hours :-)
+1 for hostPort with CNI.

[chen-anders]

I'm having issues getting HostPort working with the proper .conflist config for flannel on k8s 1.7.3 using the CNI 0.6.0-rc2 release.

This is along the lines of what I have:

{
    "name": "cbr0",
    "cniVersion": "0.3.0",
    "plugins": [
      {
        "type": "flannel",
        "delegate": {
          "isDefaultGateway": true
        }
      },
      {
        "type": "portmap",
        "capabilities": {"portMappings": true}
      }
    ]
}

I've searched over the web and there only seems to be instructions for Calico, but not flannel.

[klausenbusk]

@chen-anders What is the file extension? I have read it should be .conflist..

[chen-anders]

@klausenbusk - .conflist is correct. (Edited the original post for clarity)

Update: I was able to get it working with the v0.6.0-rc2 release downloaded from here: https://github.com/containernetworking/plugins/releases

[sonnysideup]

I guess I'm still a little confused on how one would configure hostPort. Like @c-knowles mentioned earlier, I'm seeing this problem when trying to install Datadog into our cluster.

FYI – Running kube-aws version v0.9.9-rc.2

[mumoshu]

@drywheat Hi! If you have not tried yet - Perhaps you need to include the portmap plugin in the cni config?
Ref: kubernetes/kubernetes#23920 (comment)

[cknowles]

On the datadog side they are adding/migrating to unix socket support in their next agent which will remove this as an issue and allow for better pod introspection. But it’s still a problem I’d like to resolve for other cases.

[9len]

kubernetes 1.9 includes CNI 0.6.0, which should support hostPort with flannel, via the portmap plugin.

[whereisaaron]

@9len I think it was Calico+flannel that had the hostPort issue? And this was fixed in Kubernetes 1.7. Certainly hostPort works with kube-aws today (with Calico enabled). I updated the kube-aws documentation recently. Is that the same issue you are talking about, or something different?

[9len]

I believe this was about flannel + cni, for which (as far as I was able to determine) hostPort doesn't work without cni 0.6.0 (which was added in kube 1.9)

[fengyd2018]

hostPort cannot work with kubernetes 1.9.2 and CNI 0.6.0 ?

My enviroment:
[root@worker ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

kubernetes-cni-0.6.0-0.x86_64

[whereisaaron]

Calico is CNI provider. Certainly Fannel+Calico+CNI+hostPort works in k8s 1.8.x because that is what we run. Reportly this combo works in 1.7 also - thanks to the introduction of plugin-chaining in that version, but I haven't personally tested that. There used to be a problem with hostPort but the fix for that was merged 1 Jun 2017 (via the port-forwarded plugin) and so made it into k8s 1.7.

[fengyd2018]

Flannel+CNI+HostPort cannot work?

[9len]

Based on the changelog, hostPort/flannel works with kubernetes >1.9, which includes cni >0.6.0

[fengyd2018]

[root@worker ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T10:09:24Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.2", GitCommit:"5fa2db2bd46ac79e5e00a4e6ed24191080aa463b", GitTreeState:"clean", BuildDate:"2018-01-18T09:42:01Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

It means kubernetes version is 1.9.2?

[fengyd2018]

I downloaded cni-plugins-amd64-v0.7.0-rc2.tgz from https://github.com/containernetworking/plugins/releases.
It's enough I extracts the files and copy them to /opt/cni/bin?
Anything else is needed?

[fengyd2018]

With https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml, hostPort can work

[cknowles]

So to get this working using flannel with calico, I assume we need to use https://github.com/coreos/flannel-cni. @mumoshu how much do you know about the various network fabrics? My knowledge is a bit limited so I could work on some of this but need a little guidance about what we need to add. I hit this issue again today, I have a Daemon Set with some hostPort set and need to also access that in some cases via a k8s Service. It's a weird use case but mainly due to not being able to migrate all systems at the same time. With the current flannel setup there's no way to achieve that without manual intervention as far as I can tell, but I think setting up CNI fully can help resolve it.

[cknowles]

I confirmed that enabling #1195 solves this issue, at least using the canal setup which I think should be the recommended one until we support amazon-vpc-cni-k8s in #1047.

[cknowles]

For some reason this worked in my previous tests with k8s 1.9.9 and canal self hosted but I checked again yesterday with kube-aws cb6766d and it seems hostPort is not working again. Not entirely sure why as the canal config looks fine.

[cknowles]

Nevermind, ignore my above comment - I've since found an unrelated configuration issue which made it look like this wasn't working but it was fine. I've re-confirmed all expected paths work:

• A pod using downward spec.nodeName can communicate to another pod on the same host that exposes hostPort
• A pod can communicate with a k8s Service port that points at another pod's hostPort
• A host can communicate with a pod on itself exposing hostPort via localhost
• A host can communicate with a pod on itself exposing hostPort via `hostname`