Applying managed-cluster add-on leads to null-error when using e.g. custom filter on the datastructures

[xenji]

What happened?
After applying the add-on for managed clusters, I find that the objects, null'ed by the add-on are present in the data-structures and are written to the output.

I've hat a conversation with @paulfantom about a different topic on Slack (https://kubernetes.slack.com/archives/CFFDS2Z7F/p1616409344029800) and the solution drove me into finding the issue.

Did you expect to see some different?
I would expect the add-on not to break unrelated code or force the user to introduce workarounds / null checks.

How to reproduce it (as minimally and precisely as possible):

• Take the code from here https://github.com/openshift/cluster-monitoring-operator/blob/02d1245bdebd244e340b1bac93dc0e24e42d549a/jsonnet/remove-runbook-urls.libsonnet for my custom code. It basically does the same thing, not with an annotation, but with a label. Both code snippets break for the same reason.
• I've tested kube-prometheus using the revision 4d200f2

Steps to reproduce:

1. Setup kube-prometheus via jb
2. Include the managed-cluster add-on
3. Add the code from https://github.com/openshift/cluster-monitoring-operator/blob/02d1245bdebd244e340b1bac93dc0e24e42d549a/jsonnet/remove-runbook-urls.libsonnet to the project, using the function as curry for the manifest generation
4. try to render the manifests

The result is similar to the following block, beside that it references paths on my local machine

RUNTIME ERROR: Value non indexable: *jsonnet.valueNull
	labels/addRuleLabels.libsonnet:12:12-18	function <anonymous>
	labels/addRuleLabels.libsonnet:14:10-26	object <anonymous>
	Field "kubernetes-serviceMonitorKubeControllerManager"	
	During manifestation

Environment

• Prometheus Operator version:

  4d200f29f594c90c4850848c2611f062a90ed046

Anything else we need to know?:

In a Slack conversation with @paulfantom, he discovered that this issue is related to google/jsonnet#312

Workaround:
@paulfantom worked with me to find a temporary workaround for the issue and this is the version that worked out for me. As said, this is my local version, adopted from the runbook code mentioned above.

local k8sMixinUtils = import 'github.com/kubernetes-monitoring/kubernetes-mixin/lib/utils.libsonnet';

local addLabels(rule) = rule {
  [if 'alert' in rule && ('labels' in rule) then 'labels']+: {
    namespace: 'monitoring',
    env: std.extVar('target_env'),
  },
};

{
  addLabels(o): {
    local filterRule(o) = o {
      [if (o.kind == 'PrometheusRule') then 'spec']+: k8sMixinUtils.mapRuleGroups(addLabels),
    },
    [k]: filterRule(o[k])
    for k in std.objectFields(o) if std.isObject(o[k])
  },
}

[paulfantom]

A solution would be to hide a field in https://github.com/prometheus-operator/kube-prometheus/blob/main/jsonnet/kube-prometheus/addons/managed-cluster.libsonnet#L8. Something like:

[q]:: null

Sadly this is not yet supported by jsonnet.

It is possible that the issue will present itself in every place where we remove some fields.