Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for stateful jobs and execution controller pod priority in k8s #3249

Closed
godber opened this issue Mar 31, 2022 · 3 comments · Fixed by #3252
Closed

Add support for stateful jobs and execution controller pod priority in k8s #3249

godber opened this issue Mar 31, 2022 · 3 comments · Fixed by #3252
Assignees
Labels
enhancement k8s Applies to Teraslice in kubernetes cluster mode only. pkg/teraslice

Comments

@godber
Copy link
Member

godber commented Mar 31, 2022

As an operator, I need to be able to specify a k8s pod priority so that stateful workers and execution controllers are not pre-empted. See:

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

@godber godber added enhancement k8s Applies to Teraslice in kubernetes cluster mode only. pkg/teraslice labels Mar 31, 2022
@godber godber self-assigned this Mar 31, 2022
@godber
Copy link
Member Author

godber commented Mar 31, 2022

I think this means I need to let the operator supply the name of a priority class in the terafoundation config file.

@godber
Copy link
Member Author

godber commented Apr 5, 2022

I am going to add the following Teraslice Master configuration setting:

    kubernetes_priority_class_name: {
        doc: 'Priority class that the Teraslice master, execution controller, and stateful workers should run with',
        default: undefined,
        format: 'optional_String'
    },

@godber
Copy link
Member Author

godber commented Apr 5, 2022

I had forgotten that we never implemented a first class job property for stateful to indicate that workers shouldn't be disrupted without fully restarting the job. Internally we have just been using the label feature to indicate it like this:

    "labels": {
        "stateful": true
    },

I am not sure it's appropriate to depend on label contents for a feature like this, so I will probably just implement the preemption stuff for execution controllers first. Once that's working right, I might try and extend it to include the label thing and make notes in the code that this is a bad idea to do in the long run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement k8s Applies to Teraslice in kubernetes cluster mode only. pkg/teraslice
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant