Some way to get eck-operator to create a user for a non-CRD application

[AJMansfield]

I'm currently working on migrating an existing Logstash into kubernetes. Due to the complexity of the logstash pipeline and the fact that this logstash configuration is still being maintained upstream from me, migrating that pipeline to filebeat isn't really an option.

In order to do this, I need to somehow get eck-operator to create a user and generate a set of secrets in the appropriate namespace for my logstash instance to use them to submit its logs, similar to the way setting up a Beat using the CRD creates the {name}-beat-es-ca and {name}-beat-user secrets in the corresponding namespace.

My Current Workaround Attempt

At the moment I'm attempting to use a rather hack-y workaround involving using a filebeat CRD with zero replicas to get eck-operator to create a user and relevant secrets, and then grabbing those secrets in the definition of the logstash pod.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: logstash
  namespace: myns
spec:
  type: filebeat
  version: 8.6.2
  elasticsearchRef:
    name: elasticsearch
    namespace: eck
  kibanaRef:
    name: kibana
    namespace: eck
  deployment:
    replicas: 0
---
apiVersion: v1
kind: Pod
metadata:
  name: logstash
  namespace: myns
spec:
  containers:
  - name: logstash
    image: docker.elastic.co/logstash/logstash:8.6.2
    env:
    - name: ES_HOSTS
      value: "https://elasticsearch-es-http.eck.svc:9200"
    - name: ES_USER
      value: "myns-logstash-beat-user"
    - name: ES_PASSWORD
      valueFrom:
        secretKeyRef:
          name: "logstash-beat-user"
          key: "myns-logstash-beat-user"
    volumeMounts:
    - name: config-volume
      mountPath: /usr/share/logstash/config
    - name: pipeline-volume
      mountPath: /usr/share/logstash/pipeline
    - name: elasticsearch-certs
      mountPath: /etc/logstash/certificates
      readOnly: true
  volumes:
  - name: config-volume
    configMap:
      name: logstash-config
  - name: pipeline-volume
    configMap:
      name: logstash-pipeline
  - name: elasticsearch-certs
    secret:
      defaultMode: 420
      optional: false
      secretName: logstash-beat-es-ca

Aside from this being a rather dirty hack, this also still doesn't quite work: the created user still doesn't have all of the roles that logstash needs, giving me errors like this one:

[INFO ] 2023-03-03 18:36:54.753 [[logstash]>worker1] elasticsearch - Retrying failed action {
    :status=>403,
    :action=>[
        "index",
        { :_id=>nil, :_index=>"logstash-2023.03.03", :routing=>nil },
        { "message"=>"********" }
    ],
    :error=>{
        "type"=>"security_exception",
        "reason"=>"action [indices:admin/auto_create] is unauthorized for user [myns-logstash-beat-user] with effective roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [logstash-2023.03.03], this action is granted by the index privileges [auto_configure,create_index,manage,all]"
    }} 

The description for the type in the beats.beat.k8s.elastic.co CRD indicates that the user elastic-operator creates gets granted roles based this value:

Type is the type of the Beat to deploy (filebeat, metricbeat, heartbeat, auditbeat, journalbeat, packetbeat, and so on). Any string can be used, but well-known types will have the image field defaulted and have the appropriate Elasticsearch roles created automatically.

I can't find any indication of what the set of "well-known types" is though, or if there's anything I can put here to convince it to give the logstash user the roles it needs.

Beyond that, there don't seem to be any other mechanisms for customizing this that can be done from a manifest file.

Make This its own Custom Resource?

It really seems like this should be something that's accomplished with its own custom resource, without needing to trick elastic-operator into doing it with a fake beat, and with some mechanism for customizing the resulting user roles.

[AJMansfield]

I did find this one comment on another issue that seems to indicate another possible approach: #728 (comment)
Essentially, creating the underlying object that elastic-operator uses to store the user information, and then making elastic-operator the "owner" of that secret.

This still seems like a hack-y workaround and requires additional steps to generate the password and hash, but I might try something along those lines.

[pebrc]

We fully support the Elasticsearch file realm in ECK https://www.elastic.co/guide/en/cloud-on-k8s/2.6/k8s-users-and-roles.html#k8s_file_realm

You can create custom users with the necessary roles through that. A complication you have is that your Logstash is running in a different namespace than your Elasticsearch cluster which means you will have to copy the secrets that define the users into both namespaces.

We also currently working on supporting Logstash as a first class citizen on ECK (check out the currently open PRs) which will be made available in one of the next releases.

I am therefore closing this issue.