Add ETL resolution configurations #802
Conversation
| @@ -436,6 +436,10 @@ spec: | |||
| - name: ETL_STORE_DURATION_DAYS | |||
| value: {{ (quote .Values.kubecostModel.etlStoreDurationDays) | default (quote 120) }} | |||
| {{- end }} | |||
| - name: ETL_RESOLUTION_SECONDS | |||
| value: {{ (quote .Values.kubecostModel.etlResolutionSeconds) | default (quote 60) }} | |||
There was a problem hiding this comment.
What's our process for default in the template vs. default in values?
There was a problem hiding this comment.
Good question. I provide both to be "safe", but we should decide as a team. Do you have a preference?
There was a problem hiding this comment.
It is best to provide both for common levers so people can easily read values to see the default and users who don't often update their values get a sensible default, in case they are installing from an old values.yaml
If its something hairy like a scaling parameter, it can be left off of values though IMO to avoid clutter. This feels hairy enough that it can be hidden from the majority of users and just put in the template.
Any other thoughts?
There was a problem hiding this comment.
Agreed with that. Only thing left to discuss is what the default values should be. Here, I've opted for ones that prioritize accuracy and hedge against timeouts for medium- to medium-large clusters. But a large customer might still want to make resolution 5m, or even batch 3h, to hedge further. I'd just hate to default to something that, for instance, misses all the checks pods. But open to opposing arguments.
There was a problem hiding this comment.
Agreed to discuss Monday with @dwbrown2 and @AjayTripathy
Supports opencost/opencost#719 and https://github.com/kubecost/kubecost-cost-model/pull/347
Changes
kubecostModel.etlResolutionSeconds(i.e.ETL_RESOLUTION_SECONDS) determines how high of a resolution ETL should use. Defaults to60(i.e. 1m) but should be set to300(i.e. 5m) or even600(i.e. 10m) for big clients.kubecostModel.etlMaxBatchHours(i.e.ETL_MAX_BATCH_HOURS) limits the window duration of expensive ETL queries, batching them out so that any one query will not time out. Defaults to6(i.e. 6h) but might need to be set to3(i.e. 3h) for big clients who want to keep high resolution.Testing