Skip to content
This repository has been archived by the owner on Nov 1, 2022. It is now read-only.

Replica counts changing based on time of day #2656

Closed
pcfens opened this issue Dec 3, 2019 · 7 comments
Closed

Replica counts changing based on time of day #2656

pcfens opened this issue Dec 3, 2019 · 7 comments
Labels

Comments

@pcfens
Copy link
Contributor

pcfens commented Dec 3, 2019

We have some workloads that we'd like to scale up and down based on time of day (e.g. anticipated loads, scaling non-production instances to 0 to save money after hours, etc). I don't think the fix outlined in #1686 will work because either spec.replicas (in a deployment) or spec.minReplicas/spec.maxReplicas (in an HPA) needs to change.

As I understand it, a user can adjust the number of replicas by editing a deployment directly or by adjusting the limits in an associated HPA. The mechanism might be manual (captured in git), a CronJob, or some schedule controller. I'm concerned that the automated options would be undone by the flux agent. An HPA native approach never got a lot of traction (kubernetes/kubernetes#49931)

The only fixes I can think of that keep git as a source of truth involve syncing some data back to the repository. Is there a better option?

@pcfens pcfens added blocked-needs-validation Issue is waiting to be validated before we can proceed enhancement labels Dec 3, 2019
@2opremio
Copy link
Contributor

I don't think the fix outlined in #1686 will work because either spec.replicas (in a deployment) or spec.minReplicas/spec.maxReplicas (in an HPA) needs to change.

Why not? As long as you keep the HPA in git and edit the values there, it should work.

@2opremio 2opremio added question and removed blocked-needs-validation Issue is waiting to be validated before we can proceed enhancement labels Dec 20, 2019
@pcfens
Copy link
Contributor Author

pcfens commented Dec 20, 2019

Editing in git works well when it's a manual change but not for automatic changes like scaling a test environment to 0 overnight and on weekends.

@pcfens
Copy link
Contributor Author

pcfens commented Feb 28, 2020

There are a number of options out there that will modify HPA objects or the replica count in a Deployment or StatefulSet based on the time of day.

As long as flux automation isn't turned on something like kubernetes-cronhpa-controller should work if we leave replicas blank in the deployment specification. The one downside is that we'll only be running 1 replica between the initial deployment (or manual upgrade) until the first scaling event.

Without the ability to sync something back to the git repository I'm still not sure how we can make kubernetes auto-scale workloads based on time of day, default to more than one replica, and work with a gitops workflow.

@graywolf-at-work
Copy link

Without the ability to sync something back to the git repository

Is that a problem? Why cannot the cron commit the result?

@pcfens
Copy link
Contributor Author

pcfens commented May 7, 2020

cron could commit a change, but then it would require the job be the only thing that dynamically modifies the objects and for the user to handle yaml parsing and SSH keys. The right fix for this might be the ability to selectively sync resources/fields from Kubernetes back to git (object annotations?).

@kingdonb
Copy link
Member

As Flux contrib efforts have been focused on Flux v2, we have developed a support policy page with references to how community works in our new architecture for Flux v2, with more different ways to meet and engage with us for help. If you've been following our development efforts then of course we hope you are able to upgrade, here's more info on how to find support with that: https://fluxcd.io/support/ - tl;dr: the Flux project has moved to a new repo, fluxcd/flux2

If this is still affecting you, I will be happy to reopen and/or troubleshoot with you here, or in the #flux slack channel on CNCF, but in the interest of reducing the number of open issues not directly related to supporting Flux v1 in maintenance mode, and respecting you may have moved on already, I will go ahead and close out this issue for now.

@pcfens
Copy link
Contributor Author

pcfens commented Feb 24, 2021

Hi @kingdonb- I think this is still open, but it's likely moved from a question to a feature request. I'll re-create the request with a better description in the new repo. Thanks!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

4 participants