Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Possibility to use OpenStack without lbaas (loadbalancer) #7178

Merged
merged 3 commits into from
Jul 9, 2019

Conversation

zetaab
Copy link
Member

@zetaab zetaab commented Jun 22, 2019

Some OpenStack users do have situation that they do not have lbaas resource installed at all in their setup. This PR will add very-experimental support for using OpenStack without lbaas.

Non loadbalancer deployments will generate a certificate with the 3 master floating IP addresses.
Kubeconfig will choose arbitrarily between each master.

Huge thanks to original PR author @drekle #6608

fixes #6584

/sig openstack

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. area/provider/openstack Issues or PRs related to openstack provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 22, 2019
@zetaab
Copy link
Member Author

zetaab commented Jun 22, 2019

/hold

@k8s-ci-robot k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 22, 2019
@zetaab
Copy link
Member Author

zetaab commented Jun 22, 2019

/hold cancel

@k8s-ci-robot k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 22, 2019
@zetaab zetaab force-pushed the feature/nonlb branch 4 times, most recently from 1f29ba8 to 9cf0b45 Compare June 23, 2019 09:24
@zetaab zetaab changed the title WIP: Possibility to use OpenStack without lbaas Possibility to use OpenStack without lbaas Jun 24, 2019
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 24, 2019
@zetaab
Copy link
Member Author

zetaab commented Jun 24, 2019

@drekle can you review this. Like you said this is not "production" ready when there is no loadbalancer in front of kubernetes cluster. However, I would like to make it possible to install it like this - people can use for instance external loadbalancers or metallb in front of k8s api. Actually I am currently trying to get metallb working, so we could install metallb automatically if there is no lbaas. Anyways this is first step to get support to install it like this.

After this PR I can install new kops openstack cluster using octavia (neutron-lbaas not tested) or without lbaas. Installing without lbaas is not HIGHLY AVAILABLE from outside cluster

@zetaab
Copy link
Member Author

zetaab commented Jun 24, 2019

/assign @mikesplain @justinsb

@zetaab zetaab changed the title Possibility to use OpenStack without lbaas Possibility to use OpenStack without lbaas (loadbalancer) Jun 24, 2019
@drekle
Copy link
Contributor

drekle commented Jun 24, 2019

I'll try to test this soon. On the fence about creating a production k8s cluster in this way, and if Kops should support, but open to input from others here.

@drekle
Copy link
Contributor

drekle commented Jun 25, 2019

@zetaab can we mark this as something which will resolve #6584 ?

@drekle
Copy link
Contributor

drekle commented Jun 25, 2019

I've personally tested this, it seems fine. Running multi-master with no external LB is not as bad as running a single master, which Kops unfortunately supports, so this seems okay to do.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jun 25, 2019
Copy link
Contributor

@mikesplain mikesplain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good! Thanks so much @zetaab!

/retest
/lgtm
/approve

@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikesplain, zetaab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 27, 2019
@zetaab
Copy link
Member Author

zetaab commented Jun 27, 2019

/test pull-kops-e2e-kubernetes-aws

4 similar comments
@zetaab
Copy link
Member Author

zetaab commented Jun 28, 2019

/test pull-kops-e2e-kubernetes-aws

@zetaab
Copy link
Member Author

zetaab commented Jul 1, 2019

/test pull-kops-e2e-kubernetes-aws

@zetaab
Copy link
Member Author

zetaab commented Jul 8, 2019

/test pull-kops-e2e-kubernetes-aws

@zetaab
Copy link
Member Author

zetaab commented Jul 9, 2019

/test pull-kops-e2e-kubernetes-aws

@zetaab
Copy link
Member Author

zetaab commented Jul 9, 2019

/retest

@k8s-ci-robot k8s-ci-robot merged commit 2f14693 into kubernetes:master Jul 9, 2019
@MPV
Copy link

MPV commented Mar 6, 2020

Did anyone try further setting up MetalLB (or gobetween like how KubeOne does this)?

@zetaab
Copy link
Member Author

zetaab commented Mar 6, 2020

@MPV yes I have tried it: 1) it cannot be used for kubernetes master API 2) you need remove usually port-security from neutron ports to get it working. If thats fine then you could use MetalLB, but at least for us removing all security is no-go

@zetaab zetaab deleted the feature/nonlb branch March 6, 2020 10:43
@MPV
Copy link

MPV commented Mar 6, 2020

@zetaab Ah, yes, I've seen those known issues for using MetalLB as an in-cluster external load balancer too:
https://metallb.universe.tf/installation/clouds/#metallb-on-openstack
https://metallb.universe.tf/faq/#is-metallb-working-on-openstack

My main desire here was around no 1: to have a non-OpenStack load balancer for the kubernetes masters (not load balancing services in kube), which is why I was thinking of how KubeOne does load balancing of kube masters/API-servers using gobetween (their motivation here).

I'm not yet ready to make this kind of patch to kops myself, but in case anyone else would be interested, we could take a look at how KubeOne deploys gobetween to load balance the masters:

Here's how they configure it:
https://github.com/kubermatic/kubeone/blob/v0.11.0/examples/terraform/openstack/etc_gobetween.tpl

And here how they deploy it:
https://github.com/kubermatic/kubeone/blob/v0.11.0/examples/terraform/openstack/gobetween.sh

And here's how they add the IPs of the different masters into its config:
https://github.com/kubermatic/kubeone/blob/v0.11.0/examples/terraform/openstack/main.tf#L175-L203

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/provider/openstack Issues or PRs related to openstack provider cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Install kpos in Openstack
6 participants