Operator cannot create resources in other namespaces even in cluster-wide mode

[kevinearls]

After PR #928 I have not been able to install an operator that works cluster-wide. It's possible I am not following the updated instructions correctly, but here is what I did:

kubectl create namespace observability
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/crds/jaegertracing.io_jaegers_crd.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/service_account.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role.yaml
kubectl create -n observability -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/role_binding.yaml

kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/cluster_role.yaml
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-operator/master/deploy/cluster_role_binding.yaml

kubectl create -n observability -f https://gist.githubusercontent.com/kevinearls/0da613ed3c580c5a82d9b2399600bb11/raw/a6b8b684398f8d69e1e38ed62b8d6bebd8b26586/operator.yaml

(Note the last line uses my modified version of operator.yaml)

The operator appears to install correctly, but when I do the following:

kubectl create namespace simple
kubectl create -f deploy/examples/simplest.yaml --namespace simple

I get the following errors in the jaeger operator log. (Actually I get them repeatedly, as in 1 a second.)

E0228 09:04:35.171558       1 reflector.go:123] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:96: Failed to list *v1.Secret: secrets is forbidden: User "system:serviceaccount:observability:jaeger-operator" cannot list resource "secrets" in API group "" at the cluster scope

This occurs both on minikube and on Openshift

[jpkrohling]

I have a workaround for this, but looks like the problem is underneath our operator, either at the SDK, controller-runtime or perhaps even client-go.

[jpkrohling]

There are actually a couple of problems happening here:

1. the cluster role does not include the rules from the regular role, meaning that a role binding has to be created for each namespace in which the operator should create resources
2. when this permission doesn't exist, we hit a problem like client#List with client.InNamespace ListOption requires permissions at cluster-level operator-framework/operator-sdk#2608

The problem you reported is most likely the first item, so, I'll change the title back to that and create a new one to track the second problem.

[majidazimi]

The issue still exists in latest version. I'm using chart 2.21.1 with image 1.22.1, I've set the following options:

rbac:
  create: true
  clusterRole: true

But I still see this log message:
E0525 08:30:16.437023 1 reflector.go:178] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:125: Failed to list *v1.Namespace: namespaces is forbidden: User "system:serviceaccount:jaeger-operator:jaeger-operator" cannot list resource "namespaces" in API group "" at the cluster scope

But when I add namespaces to api groups "", everything works.

This is the role that helm has generated:

---
# Source: jaeger-operator/templates/role.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jaeger-operator
  namespace: {{ .Release.Namespace }}
  labels:
    
    app.kubernetes.io/name: jaeger-operator
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - persistentvolumeclaims
  - events
  - configmaps
  - secrets
  - serviceaccounts
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - deployments
  - daemonsets
  - replicasets
  - statefulsets
  verbs:
  - '*'
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs:
  - '*'
- apiGroups:
  - monitoring.coreos.com
  resources:
  - servicemonitors
  verbs:
  - get
  - create
- apiGroups:
  - io.jaegertracing
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - extensions
  resources:
  - replicasets
  - deployments
  - daemonsets
  - statefulsets
  - ingresses
  verbs:
  - "*"
- apiGroups:
  - batch
  resources:
  - jobs
  - cronjobs
  verbs:
  - "*"
- apiGroups:
  - route.openshift.io
  resources:
  - routes
  verbs:
  - "*"
- apiGroups:
  - logging.openshift.io
  resources:
  - elasticsearches
  verbs:
  - '*'
- apiGroups:
  - jaegertracing.io
  resources:
  - '*'
  verbs:
  - '*'
- apiGroups:
  - rbac.authorization.k8s.io
  resources:
  - clusterrolebindings
  verbs:
  - '*'
- apiGroups:
  - apps
  - extensions
  resourceNames:
  - jaeger-operator
  resources:
  - deployments/finalizers
  verbs:
  - update
- apiGroups:
  - kafka.strimzi.io
  resources:
  - kafkas
  - kafkausers
  verbs:
  - '*'
- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - '*'