In-cluster building doesn't work on DigitalOcean (doks)

[edvald]

Bug

Current Behavior

Currently, in-cluster building doesn't work with DigitalOcean doks clusters.

The garden-system all deploy as normal, but the cluster is unable to pull from the in-cluster registry. The proximal cause appears to be that hostPort pods (through DaemonSets) can't be reached (connection refused), even for other services. I've tried all manner of things and am stuck on fixing this.

Expected behavior

For in-cluster building to work, same as on GKE, AKS etc.

Reproducible example

Try configuring a doks cluster environment in the demo-project and deploying the project. It will eventually fail with ImagePullBackOff because the cluster is unable to reach the in-cluster registry.

Workaround

Use the default local-docker build mode when deploying to doks clusters.

Suggested solution(s)

We need to reach out to DO to figure out why hostPort services refuse connections.

Your environment

Latest master.

[clems71]

Hey there!

Same thing on my side with a Kops provisioned cluster on AWS. Using default Kops settings. No fancy networking stuff. I'm basically getting the same error.
Everything deployed properly, cluster-init ran fine as well. Only when I want to deploy or dev, it fails with:

Error deploying backend: ImagePullBackOff - Back-off pulling image "127.0.0.1:5000/demo-project/backend:v-5b72e91a7c"

[clems71]

By investigating more and looking into the logs of the registry-proxy DaemonSet, I've found that requests have been filtered out. That's mostly due to the range parameter used on socat. Here is a sample log line I've got:

2019/06/28 08:42:51 socat[8] W refusing connection from AF=2 100.96.3.1:40244 due to range option

Initial command in registry-proxy:

socat -d TCP-LISTEN:5000,fork,range=10.0.0.0/8 TCP:garden-docker-registry.garden-system.svc.cluster.local:5000

I updated it for the moment to make it work by removing the range param (I'm sure there are some security implications I'm not aware of, but at least it makes things work):

socat -d TCP-LISTEN:5000,fork TCP:garden-docker-registry.garden-system.svc.cluster.local:5000

HTH,
Cheers

[valerauko]

I face the same issue. Sadly removing the range option didn't resolve it (or maybe I removed it from the wrong spec)

[eddiezane]

👋 Eddie from the DigitalOcean DevRel team here.

TLDR: This should be fixed with a new DOKS image shipping this week.

This is due to the currently used version of our CNI (Cilium) not supporting hostPort out of the box. A newer version adds a flag that makes enabling it easy. A new version of DOKS should be shipping this week that enables this.

I've been told you this can be used as a workaround for the time being https://github.com/snormore/cilium-portmap.

[edvald]

@clems71 Ah, we probably need to dynamically work out the correct address range. Key thing was to make sure we're not allowing outside traffic accidentally, as a side-effect of our little trickery to get the in-cluster registry going. I'll dig into this, see how we might best solve this across the board.

@eddiezane Thanks for the quick response! Once it's released, I expect I need to update my existing cluster(s)?

[eddiezane]

@edvald you should be able to upgrade your cluster to the latest patch version once it lands via the console or the automatic maintenance window. Fix should be baked into all images/minor versions.

[edvald]

@eddiezane That's awesome, thanks again for the fast response!

@clems71 your issue is something we need to fix on our side, we'll figure it out for our next patch release this week.

[clems71]

Awesome thanks!

[edvald]

@clems71 I believe #930 solves your issue. Just tried it myself on a kops cluster and seems to do the trick. It'll be in v0.10.1, so you can get rid of the workaround then .)

[clems71]

Ok duly noted! Will check ASAP. Thanks for the feedback.

…

On Sun, 7 Jul 2019 at 20:46, Jon Edvald ***@***.***> wrote: @clems71 <https://github.com/clems71> I believe #930 <#930> solves your issue. Just tried it myself on a kops cluster and seems to do the trick. It'll be in v0.10.1, so you can get rid of the workaround then .) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#877?email_source=notifications&email_token=AAQUSQ7QKD7HG3FN7BJLLPTP6I2ZZA5CNFSM4H274UH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZLQ6OY#issuecomment-509022011>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAQUSQ4CGZEOMDUBNIGEHITP6I2ZZANCNFSM4H274UHQ> .

-- *Clément JACOB*

[edvald]

@eddiezane I just checked with the latest version (1.14.3-do.0) and still have the same issue. Do you have an issue filed that we could track?

[eddiezane]

@edvald ack. Just pinged the team again.

[timoreimann]

@edvald we recently created https://github.com/digitalocean/DOKS to allow DOKS users to create issues and generally get in touch with our team. Feel free to file a bug report so that we can keep you posted on any updates. (I know some of my colleagues are already looking into the issue.)

Thanks!

[solomonope]

Hello,

I tested this with a Digital Ocean K8s cluster version 1.16.2-do.0 . I was able to build and deploy.

[edvald]

Cool! Could you take a quick look at #995 as well @solomonope?