Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kubernetes LoadBalancer race conditions and other issues #532

Closed
3 of 4 tasks
regisb opened this issue Nov 22, 2021 · 1 comment · Fixed by #539
Closed
3 of 4 tasks

Kubernetes LoadBalancer race conditions and other issues #532

regisb opened this issue Nov 22, 2021 · 1 comment · Fixed by #539
Labels
feature request New features will be processed by decreasing priority

Comments

@regisb
Copy link
Contributor

regisb commented Nov 22, 2021 •
edited
Loading

Initial conversation: https://discuss.overhang.io/t/tutor-k8s-in-production-mode-is-this-a-dns-catch-22/2119/

In this conversations, several issues were raised:

  1. A Kubernetes deployment requires a valid host name to run properly. Yet, we want to keep supporting minikube and local k8s cluster environments, as well as non-https deployments.
  2. When https is enabled, a valid Caddy service must be ready before the platform is initialised. Otherwise, initialisation will fail. This is a sort of race condition in setting up the deployments and services.
  3. When the platform is stopped, with tutor k8s stop, the LoadBalancer is deleted. As a consequence, when it is started again, its external IP will change, and DNS records will have to be updated.

We propose the following fixes:

  • Somehow, find a way to run in non-production mode on a local cluster. If not, we should not be asking the user if they want to run in non-production mode during tutor save -i.
  • Provide an easy way to create a LoadBalancer object, such as tutor k8s start caddy. This will leave time to the end user to create the right DNS records.
  • Ensure that the Caddy deployment is up prior to platform initialisation.
  • Do not delete the LoadBalancer on tutor local stop.

K8s deployment will It's ok if the k8s deployment is not exactly a 1-click installation. As mentioned in the docs, k8s deployments is aimed at operators who know what they are doing. There are so many k8s environments out there that we should not try to provide a one-size-fits-all deployment solution.

Note that at this point, I do not want Tutor to actively check that the DNS records are correct. This feature could be performed by a plugin, for example during initialisation.

cc @fghaas who initially started the conversation around these issues.
@regisb regisb added the feature request New features will be processed by decreasing priority label Nov 22, 2021
@fghaas fghaas mentioned this issue Nov 22, 2021
Closed
regisb added a commit that referenced this issue Nov 29, 2021
@regisb
feat: better support of caddy as a k8s load balancer
9dcd666 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
regisb added a commit that referenced this issue Nov 29, 2021
@regisb
feat: better support of caddy as a k8s load balancer
5850cce 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
@regisb regisb mentioned this issue Nov 29, 2021
Merged
regisb added a commit that referenced this issue Dec 8, 2021
@regisb
feat: better support of caddy as a k8s load balancer
c173b9a 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
regisb added a commit that referenced this issue Dec 14, 2021
@regisb
feat: better support of caddy as a k8s load balancer
39ca60f 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
@regisb
Copy link
Contributor Author

regisb commented Dec 14, 2021

This is resolved in the nightly branch.

@regisb regisb closed this as completed Dec 14, 2021
regisb added a commit that referenced this issue Dec 20, 2021
@regisb
feat: better support of caddy as a k8s load balancer
cc76caa 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
regisb added a commit that referenced this issue Dec 20, 2021
@regisb
feat: better support of caddy as a k8s load balancer
d8d0560 
This introduces quite a few changes to make it easier to run Caddy as a load
balancer in Kubernetes:

- Make it possible to start/stop a selection of resources with ``tutor k8s
  start/stop [names...]``.
- Make it easy to deploy an independent LoadBalancer by converting the caddy
  service to a NodePort when ``ENABLE_WEB_PROXY=false``.
- Add a ``app.kubernetes.io/component: loadbalancer`` label to the LoadBalancer
  service.
- Add ``app.kubernetes.io/name`` labels to all services.
- Preserve the LoadBalancer service in ``tutor k8s stop`` commands.
- Wait for the caddy deployment to be ready before running initialisation jobs.

Close #532.
@fghaas fghaas mentioned this issue Jan 25, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature request New features will be processed by decreasing priority
Projects
Tutor project management
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Multiple improvements to the Caddy load balancer in Kubernetes overhangio/tutor
1 participant
@regisb