fix: allow jobs and sts to have the same tolerations and nodeselectors

[alejandroEsc]

Tested this locally:

• Creates tolerations and node-selectors for all pods/jobs from a global tolerations and nodeSelectors field.
• Allow for sts to override these values if so needed

Fixes #305

[vuldin]

Looks great, thanks for getting this PR up so quickly! Also thanks for creating _statefulset-helpers.tpl, I think this approach to organizing named templates will help over time.

I've tested this change out and it works as intended. My only feedback is that the new named templates are an either/or approach when maybe they could allow an aggregate for both nodeSelector and tolerations. In other words. If values are found in both statefulset and the top-level parameter then both are used. This isn't a blocker though, we can talk about it and improve the implementation over time if we agree.

[joejulian]

My only feedback is that the new named templates are an either/or approach when maybe they could allow an aggregate for both nodeSelector and tolerations. In other words. If values are found in both statefulset and the top-level parameter then both are used. This isn't a blocker though, we can talk about it and improve the implementation over time if we agree.

This would break if the user wanted only the sts to be on one set of nodes, and all the ancillary work to run on another, ie. Jobs run on mylabel: jobs and redpanda runs on mylabel: redpanda. Merging them would create an un-matchable combination.

[joejulian]

I'm working on a PR to ensure that the jobs and statefulset pods use distinct labels so as not to have jobs match the statefulset selector. That PR needs merged first.

[RafalKorepta]

@alejandroEsc Can you add test that uses nodeSelector and tolerations? You could annotate kind workers to achieve that.

[alejandroEsc]

tolerations

I know i can use the labels on kind workers. What do you mean by "use annotations"?

[RafalKorepta]

I wrongly write annotate. I should write label them