kubecfg show does not always return stable output

[wojciechka]

I have noticed that with more complex kubecfg configuration, it does not return stable output (i.e. same output for exactly same input), which is what jsonnet does.

Unfortunately I can't provide a snippet that allows reproducing the issue at this time.

[anguslees]

I think this was fixed in #178, since that change uses "dependency" order and then falls back to alphabetical order (alpha sort on namespace/name/kind), which should be stable for any given input. Previously that alpha fallback wasn't there, so objects with equal "dependency tier" would be reported in indeterminate order.

[anguslees]

This just hurt bitnami-labs/sealed-secrets#99

#178 never fixed this for sort only other subcommands. The sort order is/was fixed to "jsonnet result order" which should be stable, but not easily predictable.

The downside of a dependency sort on show is that it requires schema information (and thus access to cluster discovery API). I will make a full dependency sort conditional, defaulting to the current behaviour (only jsonnet-order, not full dependency sort).

[primeroz]

Hi,

Was there any news about this ?

Our problem is that after a show we commit the yaml generated into a git repo, but is impossible to check the git diff since the order of resources is "always different" at every kubecfg show

Is it possible to even jsut get a flag to do alphabetical order on show ?

[anguslees]

@primeroz hrm, I thought I had improved this. Which version are you using?

[primeroz]

I am using 0.9.1

I will try and get a proof of concept example and post it here

[primeroz]

Example:

I have a jsonnet to generate some basic resources for a namespace , i run it 3 times in a row and got 3 different ordering

 $ kubecfg show namespace-default.jsonnet | egrep -B1 "^kind"
apiVersion: v1
kind: Namespace
--
apiVersion: v1
kind: ResourceQuota
--
apiVersion: v1
kind: ResourceQuota
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
--
apiVersion: v1
kind: LimitRange
--
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy



 $ kubecfg show namespace-default.jsonnet | egrep -B1 "^kind"
apiVersion: v1
kind: ResourceQuota
--
apiVersion: v1
kind: ResourceQuota
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
--
apiVersion: v1
kind: LimitRange
--
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
--
apiVersion: v1
kind: Namespace
 

$ kubecfg show namespace-default.jsonnet | egrep -B1 "^kind"
apiVersion: v1
kind: LimitRange
--
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
--
apiVersion: v1
kind: Namespace
--
apiVersion: v1
kind: ResourceQuota
--
apiVersion: v1
kind: ResourceQuota
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
--
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding

[primeroz]

@anguslees FYI i just tried with latest and the problem is still there

[primeroz]

I finally realized that by wrapping all my resources in a "kube.List" the output is now ordered and consistent between runs.

Also kubecfg is only exposing the resources inside the list unwrapping them from the list so that makes it even better

For anyone else reading this, check this example

[bigdrum]

This is really a blocker for my gitops setup. I wonder if the proposed fix makes sense.

[bigdrum]

My PR wouldn't work for the general case because of breaking the dependency order. But I'm feeding that file to Argo CD, which seems smart enough to sync things with reasonable order.