Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support file realm in the Elasticsearch spec #728

Closed
sebgl opened this issue May 3, 2019 · 43 comments · Fixed by #2682
Closed

Support file realm in the Elasticsearch spec #728

sebgl opened this issue May 3, 2019 · 43 comments · Fixed by #2682
Assignees
Labels
>feature Adds or discusses adding a feature to the product v1.1.0

Comments

@sebgl
Copy link
Contributor

sebgl commented May 3, 2019

This would allow users to inject their own users into the file realm.
The ES spec would contain the username and a reference to the password secret.
(or maybe just a reference to the secret that contains both usernames and password hashes?)


This issue explores a lot of options, we finally agreed on providing a way to setup secrets containing the "raw" file realm content as described in Elasticsearch docs:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.6.0
  auth:
    roles:
    - secretName: my-roles-1
    fileRealm:
    - secretName: my-filerealm-secret

---

# File realm in ES format (from the CLI or manually assembled)
kind: Secret
apiVersion: v1
metadata:
  name: my-filerealm-secret
stringData:
  users: |-
    rdeniro:$2a$10$BBJ/ILiyJ1eBTYoRKxkqbuDEdYECplvxnqQ47uiowE7yGqvCEgj9W
    alpacino:$2a$10$cNwHnElYiMYZ/T3K4PvzGeJ1KbpXZp2PfoQD.gfaVdImnHOwIuBKS
    jacknich:{PBKDF2}50000$z1CLJt0MEFjkIK5iEfgvfnA6xq7lF25uasspsTKSo5Q=$XxCVLbaKDimOdyWgLCLJiyoiWpA/XDMe/xtVgn1r5Sg=
  users_roles: |-
    admin:rdeniro
    power_user:alpacino,jacknich
    user:jacknich

---

# Roles in ES format
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-1
stringData:
  roles.yml: |-
    click_admins:
      run_as: [ 'clicks_watcher_1' ]
      cluster: [ 'monitor' ]
      indices:
      - names: [ 'events-*' ]
        privileges: [ 'read' ]
        field_security:
          grant: ['category', '@timestamp', 'message' ]
        query: '{"match": {"category": "click"}}'
    another_role:
      cluster: [ 'all' ]

Let's keep the door open for higher-level abstractions (eg. one secret per user with username and password fields) in the future.

@sebgl sebgl added the discuss We need to figure this out label May 3, 2019
@nkvoll
Copy link
Member

nkvoll commented May 23, 2019

Do we have an overview over the use cases where we think the file realm will be required, and the native realm (or other realms) would not be suitable?

@thbkrkr
Copy link
Contributor

thbkrkr commented Jul 30, 2019

For now, we have had very few cases:

  • there was two requests to add custom roles to roles.yml using k8s resources
  • and two other requests to set a custom password.

Custom roles can be set using the API.
A custom password for the Elastic user can only be defined at this time by using a workaround.

My feeling is that it is a "nice to have" with a low priority for the moment.

@psalaberria002
Copy link

psalaberria002 commented Sep 12, 2019

The ability to add roles and users via files or config to the CRD is very much wanted.

I am trying to monitor the cluster using the Prometheus exporter, and it requires cluster:monitor and index:monitor/stats privileges.

As a workaround (a very ugly one) I added this block to the nodeGroup config:

        prometheus.indices: false
        xpack.security.authc:
          anonymous:
            username: anonymous_user
            roles: elastic_internal_probe_user # role contains cluster:monitor
            authz_exception: true

I could not find any existing role that covers index:monitor/stats, so I had to disable that feature in the plugin.

Fetching metrics is a very common scenario (I would expect), so I hope a more native solution can be implemented in the operator.

Enjoying the operator otherwise ;) Thanks.

@pebrc
Copy link
Collaborator

pebrc commented Sep 16, 2019

Suggested: check the current implementation is sufficient (can create user, can create/define roles) and document it.

@thbkrkr
Copy link
Contributor

thbkrkr commented Oct 21, 2019

ECK does not prevent the normal use of the native realm.

By enabling the user native authentication:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: z
spec:
  version: 7.4.0
  nodeSets:
  - name: master
    count: 1
    config:   
      xpack.security.authc.realms.native.native1.order: 0

Users and roles can be created using the security APIs:

> es() {
  local uri=$1 && shift
  curl -sku $AUTH -H 'Content-type: application/json' https://$IP:9200$uri $@
}

> es /_security/role/myrole -d '{
  "cluster": ["monitor"],
  "applications": [{
    "application": "kibana-.kibana",
    "privileges": ["all"],
    "resources": ["*"]
  }]
}'
{"role":{"created":true}}

> es /_security/user/myuser -d '{
  "roles": ["myrole"], 
  "password": "buoPM@v<k7b{0rks[90Bq3gUG<n6"
}'
{"created":true}

Documentation pages:

@charith-elastic
Copy link
Contributor

It looks like adding any realm configuration disables the native realm (unless it is explicitly created). Users can still use the APIs and the UI to create users but they can't authenticate to the cluster using those credentials. See #2036 and the follow-up #2037.

@pebrc
Copy link
Collaborator

pebrc commented Nov 12, 2019

Users can use the file realm today from ECK with some restrictions, most notably there is no way to define new roles in roles.yaml at the moment. What is possible is to define a custom user using built-in roles.

In order to do that one needs to create a secret of the following form:

apiVersion: v1
data:
  name: $USERNAME_BASE64
  passwordHash: $BCRYPT_BASE64
  userRoles: $COMMA_SEP_ROLES_BASE64
kind: Secret
metadata:
  labels:
    common.k8s.elastic.co/type: user
    elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME
  name: custom-user-cd2tccd654
  ownerReferences:
  - apiVersion: elasticsearch.k8s.elastic.co/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Elasticsearch
    name: $CLUSTER_NAME
    uid: $CLUSTER_RESOURCE_UID
type: Opaque

When generating the Bcrypt hash it is important to use a bcrypt version that produces $2a$ hashes and not one of the newer variants, see: https://en.wikipedia.org/wiki/Bcrypt#Versioning_history

This is assuming Elasticsearch is configured to use the default bcrypt password hashing: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#password-hashing-algorithms

@psalaberria002
Copy link

Any plans for adding custom roles via Kubernetes objects? I understand all this is possible to do by using the ES API, but I would prefer to do this natively via Kubernetes.

We have many clusters, so automation is a must. Writing a sidecar or extra service for role and user management seems unnecessary.

@thbkrkr thbkrkr removed their assignment Dec 18, 2019
@sebgl
Copy link
Contributor Author

sebgl commented Feb 3, 2020

We decided to implement support for the file realm (roles & users), targeting ECK version 1.1.0.

@pebrc pebrc added the v1.1.0 label Feb 4, 2020
@sebgl
Copy link
Contributor Author

sebgl commented Feb 4, 2020

So before digging into the implementation I reminded myself the way things currently work.

In Elasticsearch

3 files, on each node, control the file realm.

  • users:
alpacino:$2a$10$cNwHnElYiMYZ/T3K4PvzGeJ1KbpXZp2PfoQD.gfaVdImnHOwIuBKS
jacknich:{PBKDF2}50000$z1CLJt0MEFjkIK5iEfgvfnA6xq7lF25uasspsTKSo5Q=$XxCVLbaKDimOdyWgLCLJiyoiWpA/XDMe/xtVgn1r5Sg=
  • users_roles:
admin:rdeniro
power_user:alpacino,jacknich
user:jacknich
  • roles:
click_admins:
  run_as: [ 'clicks_watcher_1' ]
  cluster: [ 'monitor' ]
  indices:
    - names: [ 'events-*' ]
      privileges: [ 'read' ]
      field_security:
        grant: ['category', '@timestamp', 'message' ]
      query: '{"match": {"category": "click"}}'

In ECK

We aggregate those 3 files a single secret mounted in all Elasticsearch nodes: elasticsearch-sample-es-xpack-file-realm. It contains the raw bytes of each file.

This file is generated from:

  • Users hardcoded in ECK:

    • elastic-internal, the ECK user
    • elastic-internal-probe, the readiness probe user
    • elastic-internal-keystore, I think this one should not exist anymore (TODO issue)
    • elastic, generated for the end-user
  • Users dynamically created:

    • <kibana-user>, generated for Kibana to reach Elasticsearch, if an association exists
  • Roles (hardcoded in ECK):

elastic_internal_keystore_user:
  cluster:
  - all
elastic_internal_probe_user:
  cluster:
  - monitor
  • Users-roles mapping hardcoded in ECK:
elastic_internal_keystore_user:elastic-internal-keystore
elastic_internal_probe_user:elastic-internal-probe
superuser:elastic,elastic-internal
  • Dynamic users-roles mapping:
kibana_system:kb-ns-kibana-sample-kibana-user

We have an internal mechanism in place to generate the Kibana user automatically by listing secrets labeled with:

common.k8s.elastic.co/type: user
elasticsearch.k8s.elastic.co/cluster-name: $CLUSTER_NAME

Those secrets are expected to contain the user name, a password hash and a list of roles for the user, as outlined by @pebrc above.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 4, 2020

I'm trying to come up with different ways things can be exposed.

tl;dr of the examples below.

There are several alternatives:

  • Storage: in the Elasticsearch CRD vs. in a Secret vs. in a ConfigMap
  • Reference: not required vs. explicit (secretName) vs. implicit (labels)
  • Users-roles binding: specified with the users or decoupled from the users definition
  • Password: only specify the hash vs. can also optionnaly specify the cleartext password that ECK hashes

Roles

They can be defined in yaml format:

click_admins:
  run_as: [ 'clicks_watcher_1' ]
  cluster: [ 'monitor' ]
  indices:
    - names: [ 'events-*' ]
      privileges: [ 'read' ]
      field_security:
        grant: ['category', '@timestamp', 'message' ]
      query: '{"match": {"category": "click"}}'

I see several options to specify them.

  1. Inline roles in the Elasticsearch spec
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    roles:
        click_admins:
        run_as: [ 'clicks_watcher_1' ]
        cluster: [ 'monitor' ]
        indices:
        - names: [ 'events-*' ]
            privileges: [ 'read' ]
            field_security:
                grant: ['category', '@timestamp', 'message' ]
            query: '{"match": {"category": "click"}}'

Pros: does not involve extra resources. Uses the same convention as the config section.
Cons: cannot be easily shared across multiple Elasticsearch resources.

In the CRD definition, we can either:
a. embed the roles schema (yaml format) in the Elasticsearch spec. Pros: the API schema is clearly defined. Cons: this schema may evolve with Elasticsearch versions, but it's impossible to map ECK CRD versions to distinct ES versions.
b. consider it's just abstract yaml content that we pass along to Elasticsearch. Pros: we don't care about the roles schema, that's the user responsibility to put the right things at the right place. Cons: it's harder to manipulate the CRD API and to catch any syntactic error.

  1. Explicitly reference roles stored in ConfigMaps
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    roles:
        - configMapName: click_admins_roles
        - configMapName: another_one
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: click_admins_roles
data:
  # data here

Pros: we're consistent with the way we handle secureSettings. Role resources can be shared across several Elasticsearch resources in the same namespace.
Cons: Additional resources are required. SecureSettings use Secrets, not ConfigMaps, so it's not exactly consistent.

Data can have several representations there:

a. Key is the role name, value is the raw yaml content

kind: ConfigMap
apiVersion: v1
metadata:
  name: click_admins_roles
data:
  click_admins: |-
    run_as: [ 'clicks_watcher_1' ]
    cluster: [ 'monitor' ]
    indices:
    - names: [ 'events-*' ]
    privileges: [ 'read' ]
    field_security:
        grant: ['category', '@timestamp', 'message' ]
    query: '{"match": {"category": "click"}}'
  another_role:
    cluster: ['all']

Pros: feels like native use of the key/value format.
Cons: to be generated from a file, this configmap needs each role to be defined in a separate file.

b. Key is irrelevant, value is the raw roles file content

kind: ConfigMap
apiVersion: v1
metadata:
  name: click_admins_roles
data:
  roles:  |-
    click_admins:
        run_as: [ 'clicks_watcher_1' ]
        cluster: [ 'monitor' ]
        indices:
        - names: [ 'events-*' ]
        privileges: [ 'read' ]
        field_security:
            grant: ['category', '@timestamp', 'message' ]
        query: '{"match": {"category": "click"}}'
    another_role:
        cluster: ['all']

Pros: ConfigMap can be generated from a roles file directly, as users would have to write when using on-prem Elasticsearch.
Cons: does not really make use of the key/value format.

  1. Explicitly reference roles stored in Secrets

Same as option 2, with same data format alternatives, expect this time we use Secrets instead of ConfigMaps:

spec:
  version: 7.5.0
  count: 3
  fileRealm:
    roles:
        - secretName: click_admins_roles
        - secretName: another_one
---
kind: Secret
apiVersion: v1
metadata:
  name: click_admins_roles
data:
  # data here (base64)

Pros: useful if users don't want roles to be public information?
Cons: base64 encoding generally makes things harder

  1. Implicitly reference ConfigMaps through labels

Instead of an explicit reference in the Elasticsearch CRD, we could use a labeling convention. All ConfigMaps labeled with common.k8s.elastic.co/type: elasticsearch-roles (or another one) end up being considered as roles to reconcile.

spec:
  version: 7.5.0
  count: 3
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: click_admins_roles
  labels:
    common.k8s.elastic.co/type: roles
data:
  # data here

Pros: keep the Elasticsearch CRD minimal.
Cons: not explicit, and does not match the way we explicitly reference secure settings secrets.

  1. Implicitly reference Secrets through labels

Same as 4) but use Secrets instead of ConfigMaps.

Users

  • Storage: it feels natural to have them defined in Secrets, but since only their hash is required it does not have to.
  • References: as described for roles above, Secrets can be either explicitly referenced in the Elasticsearch spec (similar to how we handle secureSettings), or implicitly through labeling.
  • Hashes: Technically, we only need the hash of the user passwords. It can be a bit tricky for end-users to setup though, especially combined with base64 encoding. We may want to optionnaly allow them to specify the full password so ECK takes care of computing the hash?
  • User-roles combination: we may want to store the user/roles mapping close to the user. Or decoupled in a separate section, which is closer to the Elasticsearch documentation.

Some options below.

  1. Multiple users per Secret, key=user & value=hash
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    users:
    - secretName: my-es-users
    - secretName: my-other-users
---
kind: Secret
apiVersion: v1
metadata:
  name: my-es-users
data:
  custom-username1: <base64-version-of-the-hash>
  custom-username2: <base64-version-of-the-hash>

Pros: pretty close to the way users are defined in Elasticsearch users file.
Cons: needs another place to define users/roles binding. Can only specify the hash, not the full password.

  1. One user per Secret, with multiple fields in the Secret
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    users:
    - secretName: my-es-user
    - secretName: my-other-user
---
kind: Secret
apiVersion: v1
metadata:
  name: my-es-user
data:
  name: <username-base64>
  passwordHash: <hash-base64>
  password: <cleartext-password-base64> # <--- optional
  roles: <role1,role2 - base64> # <--- (or defined elsewhere)

Pros: allows us to specify the user roles, and maybe to compute the hash ourselves.
Cons: cannot have a single secret with multiple users.

  1. User-roles decoupled from users
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    users_roles:
      user_a: role1,role2,role3
      user_b: role3

Alternatives: can reference a ConfigMap instead, or a secret.

  1. Users in the Elasticsearch spec directly
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    users:
      user_a: <hash of the password>
      user_b: <hash of the password>
    users_roles:
      user_a: role1,role2,role3
      user_b: role3

Pros: everything in one place. Makes sense if roles are also defined this way.
Cons: even though a cleartext hash should be secure, it still feels a bit strange. Cannot reuse users across multiple ES resources.

  1. All-in-one as a Secret
spec:
  version: 7.5.0
  count: 3
  fileRealm:
    secretName: es-file-realm
---
kind: Secret
apiVersion: v1
metadata:
  name: es-file-realm
data:
  users: <content of the users file, base64>
  roles: <content of the roles file, base64>
  users_roles: <content of the users_roles file, base64>

Pros: very close to Elasticsearch native format, nothing new to learn here for advanced ES users.
Cons: we do not facilitate things for k8s users that are not used to Elasticsearch.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 4, 2020

The more I think about it, the more I like the very last option: just use a single secret that contains the 3 required files (raw content), as described in the Elasticsearch documentation. No magic at all. We just point users to the official doc.

@barkbay
Copy link
Contributor

barkbay commented Feb 4, 2020

IMHO everything related to the users should be in a Secret if it's possible (incl. hashes, e.g. on Linux they are in a shadow file)
This means that I would avoid to put this kind of information in a ConfigMap or in the Elasticsearch spec.

I agree that Secrets are not easy things, stringData can ease the pain when creating them and kubectl plugins when reading.

My concern with the all in one approach is that it does not easily allow the user to reuse the roles definitions ... but I'm not sure it is really big deal either.
May be we could allow several secrets to be referenced in the spec:

spec:
  version: 7.5.0
  count: 3
  fileRealm:
    - secretName: es-users-file-realm
    - secretName: es-roles-file-realm
---
kind: Secret
apiVersion: v1
metadata:
  name: es-users-file-realm
data:
  users: <content of the users file, base64>
  roles: <content of the roles file, base64>
---
kind: Secret
apiVersion: v1
metadata:
  name: es-roles-file-realm
data:
  users_roles: <content of the users_roles file, base64>

This way es-roles-file-realm can be reused across several clusters in the namespace.

@pebrc
Copy link
Collaborator

pebrc commented Feb 4, 2020

I like the all-in-one approach too, and I agree with @barkbay that allowing to specify multiple secrets would be good and also consistent with what we do for secure settings.

My only concern is that we cannot offer any convenience for users when it comes to password hashing. But maybe that is acceptable.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 6, 2020

Let's move forward with the raw file content merged from multiple secrets.

I think we can maybe implement helpers around password hashing in a second time.
For example, by adding an additional users_cleartext section in the secret?

kind: Secret
apiVersion: v1
metadata:
  name: es-users-file-realm
stringData:
  # users_cleartext will be internally derived into 'users' by ECK, with a hash of the provided passwords
  # (the current secret is kept unmodified by ECK)
  users_cleartext: |-
    username1:cleartextpassword

@pebrc pebrc removed the discuss We need to figure this out label Feb 6, 2020
@sebgl sebgl added the >feature Adds or discusses adding a feature to the product label Feb 6, 2020
@sebgl
Copy link
Contributor Author

sebgl commented Feb 6, 2020

In case we ever want to support more security realms in the future, I think we should add an extra realms layer in the CRD:

spec:
  version: 7.5.0
  count: 3
  realms:
    file:
    - secretName: es-users-file-realm
    - secretName: es-roles-file-realm

It could also make sense to have this under a security section, but we already have TLS stuff expressed in other places.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 7, 2020

We discussed this again with @thbkrkr and are wondering how to best handle users passwords.

The main use case for this feature seems to be driven by GitOps-like operations. You have your manifest stored somewhere in Git and you just apply it to auto-deploy Elasticsearch, and auto-create any users you want. I can then imagine those users passwords are mounted into Pods of other applications running in K8s, or used by applications running outside the Kubernetes cluster.

Being able to specify the hash is nice if you know how to compute it, and if you already have another way to store the password somewhere (and maybe mount it where necessary).

However I feel like we cannot ignore the case where you'd like users to be created and their password mounted into other pods, but you don't actually care that much about specifying the password itself. Similar to how we auto-create the elastic user with a random password today.

I think there are 3 different ways we can deal with users passwords:

  1. Provide their hash
  2. Provide their password and rely on ECK to compute the hash
  3. Request a random password to be created

Number 3 seems important to me so should probably be considered in the CRD design. It would require ECK to manipulate 2 secrets:

  • the internal file realm secret containing a hash
  • a public-facing secret containing the generated password

Not sure how to best express this through the CRD.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 10, 2020

I'm fiddling with the following CRD schema:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.5.0
  realms:
    file:
      content:
      - secretName: my-file-realm
      - secretName: my-file-realm-2
      users:
      - secretName: my-user
      - secretName: my-user2
---
kind: Secret
apiVersion: v1
metadata:
  name: my-file-realm
stringData:
  users: <content of the users file>
  roles: <content of the roles file>
  users_roles: <content of the roles file>
---
kind: Secret
apiVersion: v1
metadata:
  name: my-user
stringData:
  name: my-user-name
  roles: role1,role2,role3 # optional
  passwordHash: # optional
  password: # optional

It spits the file realm spec into 2 sections:

  • content contains the raw file realm content (users, users_roles, roles.yml) - not sure about the name
  • users is a higher-level abstraction to setup users in the file realm. The referenced secrets have the same schemas as the internal association secrets we generate.

Secrets referenced in the users section must contain a name, and may include a comma-separated list of roles. The password can be handled in one of those 3 ways:

  • if passwordHash is specified, use it to populate the file realm
  • else if password is specified, use a bcrypt hash of its value to populate the file realm
  • else if neither password nor passwordHash are specified, ECK will update the secret with a randomly generated password and matching hash

I think it would give some additional flexibility and ease of use for user management, but has the following drawbacks:

  • there are 2 different ways to declare users and their roles (through content and users).
  • retrieving a user's password can be done by accessing the ["password"] field in the secret. Whereas retrieving the elastic default user password can be done by accessing the ["elastic"] field in the secret. We could address this by also including a "password" field in the elastic user secret so it looks similar (keeping the backward-compatible "elastic" field).

@charith-elastic
Copy link
Contributor

Few things to consider as part of this design are:

It's useful in certain scenarios to be able to access the password for a certain account by mounting that secret in another pod. If plaintext passwords are being considered, we should think about a key-value structure for the secret that does not hinder such usage.

There have been some questions about changing the passwords of file realm users after the cluster is created. Currently this can be done manually in a disruptive way with the following steps:

  • edit the [cluster]-es-*-user secret and replace the entry with the new base64 encoded password
  • delete the [cluster]-es-xpack-file-realm secret and wait for it to be recreated by the operator (the reconciliation logic only compares usernames and the not the password hashes to see if the secret needs to be updated)
  • restart the Elasticsearch nodes so that they pick up the new user file

It would be good to have a simpler and safer process for changing passwords as that is a reasonable thing that users might want to do on a fairly regular basis.

Perhaps its also worth thinking a little bit more about the request in #2408 and create a design that does not explicitly rely on secrets only. I have seen environments where ephemeral secrets from Vault are injected into pods by a sidecar as a file volume instead of a secret.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 10, 2020

We discussed this topic on a meeting with @thbkrkr and @barkbay and came with the following design, pretty close to the one expressed in this comment, but with a clear separation between roles and users:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.5.0
  realms:
    file:
      roles:
      - secretName: my-roles-1
      - secretName: my-roles-2
      users:
      - secretName: my-user-1
      - secretName: my-user-2
---
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-1
stringData:
  click_admins: |-
    run_as: [ 'clicks_watcher_1' ]
    cluster: [ 'monitor' ]
    indices:
    - names: [ 'events-*' ]
      privileges: [ 'read' ]
      field_security:
        grant: ['category', '@timestamp', 'message' ]
      query: '{"match": {"category": "click"}}'
  another_role: |-
    cluster: [ 'all' ]
---
kind: Secret
apiVersion: v1
metadata:
  name: my-user-1
stringData:
  name: my-user-name
  roles: role1,role2,role3 # optional
  passwordHash: # optional
  password: # optional
  • There are 2 sets of secrets: roles and users.
  • The secret for roles includes a map of yaml roles as defined in this doc
  • The secret for users matches the schema we already use for our "internal" association users.
  • We can keep the behaviour where password and passwordHash are auto-generated by the operator if not specified by the user.
  • Those user secrets can be mounted into other Pods, which especially care about the name and password fields.
  • ECK will automatically trigger a reconciliation of the [cluster]-es-xpack-file-realm on any change in the referenced secrets. I think this addresses your concern about changing passwords @charith-elastic.
  • We can adapt the existing schema of the elastic user to match the new user secret schema (maybe in a second step? Also we should do it in a non-breaking way, ie. preserving the current elastic entry in the secret).
  • We choose to explicitly reference secrets in the Elasticsearch manifest instead of using implicit labels with the cluster name on the secrets. Mostly because that's the mechanism we already have in place to deal with secure settings.

@charith-elastic
Copy link
Contributor

Thanks, this addresses most of my concerns. Couple of things that stood out to me are:

  • It looks like the proposed schema encourages creating a secret per user. Does this mean that the operator will create at least 5 secrets per Elasticsearch resource for the operator-defined users? It's certainly a cleaner approach but we already create 10+ secrets per Elasticsearch resource and this would increase that number even more. It could become a problem at scale if the operator is managing hundreds of Elasticsearch clusters as the number Kubernetes objects and watches it needs to manage will increase dramatically. It would also be a bit difficult for a human to deal with many secrets as well.
  • Any thoughts on supporting volumes (or a better method) rather than insisting on secrets for storage?

@sebgl
Copy link
Contributor Author

sebgl commented Feb 10, 2020

It looks like the proposed schema encourages creating a secret per user

Indeed. I'm wondering if we could setup, additionally, multiple users in a single secret. Same schema as above but one level deeper basically:

kind: Secret
apiVersion: v1
metadata:
  name: my-user-1
stringData:
  my-user-name:
    name: my-user-name
    roles: role1,role2,role3 # optional
    passwordHash: # optional
    password: # optional

The downside is that it becomes harder to mount this into Pods. But may be useful for our internal users?

Any thoughts on supporting volumes (or a better method) rather than insisting on secrets for storage?

That seems pretty hard to do. I understand the need for mounting eg. Certificates in a Pod through a special kind of volume. But I don't really see a way ECK could "reconcile" multiple users from multiple volumes (?). Would we need to mount these volumes in the ECK Pod?

@sebgl
Copy link
Contributor Author

sebgl commented Feb 10, 2020

Something does not work quite well with the idea of auto-creating passwords if not specified:

  1. User creates a secret without a password defined
  2. ECK generates that password and updates the secret accordingly
  3. User updates the same manifest applied during 1): the password field is reset to empty.

I think as a general rule of thumb we should not modify anything in a user-provided secret (except, maybe, annotations).

@charith-elastic
Copy link
Contributor

It looks like the proposed schema encourages creating a secret per user

Indeed. I'm wondering if we could setup, additionally, multiple users in a single secret. Same schema as above but one level deeper basically:

kind: Secret
apiVersion: v1
metadata:
  name: my-user-1
stringData:
  my-user-name:
    name: my-user-name
    roles: role1,role2,role3 # optional
    passwordHash: # optional
    password: # optional

The downside is that it becomes harder to mount this into Pods. But may be useful for our internal users?

Nesting is not great and makes it tricky to mount the secret somewhere else as you say. Does Kubernetes even allow nested objects in the stringData field? I reckon it would need to be encoded in some way.

Any thoughts on supporting volumes (or a better method) rather than insisting on secrets for storage?

That seems pretty hard to do. I understand the need for mounting eg. Certificates in a Pod through a special kind of volumes. But I don't really see a way ECK could "reconcile" multiple users from multiple volumes (?). Would we need to mount these volumes in the ECK Pod?

I haven't thought about this a lot. Off the top of my head, it could be either:

(a) Let users take care of mounting the files themselves and just provide us with a file path

realms:
  file:
    users:
       file: /mnt/path/to/user.yaml

(b) Let users specify a volume array and let the operator merge those into the pod volumes array and mount them into the container at a pre-defined path.

realms:
  file:
    users:
      - name: userfile1
        gitRepo:
          directory: /passwords
          repositroy: git.url
          revision: xxxx
      - name: userfile2
        secret:
          secretName: mysecret

@sebgl
Copy link
Contributor Author

sebgl commented Feb 10, 2020

Does Kubernetes even allow nested objects in the stringData field? I reckon it would need to be encoded in some way.

Yes, that would just be a []byte anyway. Up to us to decide whether we use a plaintext encoding (\n separated) or something like JSON or YAML.

re. mounting something other than secrets: in any case, it looks like something we could add later on, as another reference at the same level as secretName. I think we can delay that to the future in another issue, if relevant to multiple users.

@charith-elastic
Copy link
Contributor

it looks like something we could add later on, as another reference at the same level as secretName

That would be very awkward (and likely require a breaking change to the CRD) if we do find that we need to support sources other than secrets. I'd be inclined to at least follow a schema that can be easily extended to other sources even if the initial implementation only supports secrets.

@sebgl
Copy link
Contributor Author

sebgl commented Feb 12, 2020

Some updates:

About the CRD format

@charith-elastic I get your point and I think we're on the same page. I like your option b), but I think for now we should maybe not reuse k8s volume array type directly, and instead come up with our own that we may later on extend with the k8s volume array type? My only concern is the consistency with how we define secureSettings which don't have the extra secret layer. That's why I had in mind we would eventually have something like this (the only difference is that secretName is not embedded in another secret field):

realms:
  file:
    users:
      - name: userfile1
        gitRepo:
          directory: /passwords
          repositroy: git.url
          revision: xxxx
      - name: userfile2  #<--- we don't really need a name here?
        secretName: mysecret

I wish we had that secret field in secureSettings already.

Native vs. File realm

We talked offline about how we should maybe expose the native realm instead of the file realm because Elasticsearch documentation mentions the file realm is mostly for fallback/recovery. Discussing this further with the Elasticsearch team, we concluded there is no particular concern in having ECK rely on the file realm. It perfectly fits the use case here.

Auto-generating passwords

Because of this problem it's pretty hard for us to auto-generate passwords in a user-provided secret. Some alternatives:

  • create the secret ourselves, which means username and roles must be specified in the CRD
  • generate the password in another secret

Both do not feel right to me.
Some options out there to auto-generate passwords outside the scope of ECK:

My current feeling is that it would be nice to auto-hash the provided password to the right format (it does not require us to update the user's secret content). And maybe it's OK to not have the password generation feature.

@jasontedor
Copy link
Member

We talked offline about how we should maybe expose the native realm instead of the file realm because Elasticsearch documentation mentions the file realm is mostly for fallback/recovery.

We're working on getting that removed from the documentation, the caveat is not necessary.

I am double checking that the plan here is to use the elasticsearch-users tool to manage the file realm though? We do want to discourage direct manipulation of those files and would prefer that you use the tool since we support it and provide compatibility guarantees with it. If you were to go the route of managing the hash directly, I assume that you would need support in the tool for passing in a hash instead of a password that the tool computes the appropriate (bcrypt|pbkdf2) hash of?

@sebgl
Copy link
Contributor Author

sebgl commented Feb 13, 2020

I am double checking that the plan here is to use the elasticsearch-users tool to manage the file realm though?

Interesting, thanks for bringing that up. No, I think we did not plan to use it.

Internally, ECK already relies on the file realm for "internal" users (the default elastic user, Kibana and APM users, the readiness probe user). We generate the file realm in ECK code without relying on the elasticsearch-users tool. The file format seems simple enough to not bother with invoking a separate CLI and JVM in ECK container, or in every Elasticsearch container. The file realm itself is stored as a Kubernetes secret, automatically mounted at the right place (and auto-reloaded) on all Elasticsearch Pods. Having ECK rely on the elasticsearch-users tool seems much more complicated to me. There might be a simple way for us to use it that we did not consider though.

For end-users (the purpose of this issue), so far 2 alternatives stand out:

  1. Users provide the raw Elasticsearch file realm files (users, users_roles, roles.yml) as a Kubernetes secret (described in this comment). The way they generate these files is not ECK's concern. ECK would just merge the content of the files with its internal one. In practice, I think users will manually create the roles.yml file. They can use the elasticsearch-users tool to generate users. To do so, they need an environment where Java is installed, and where an Elasticsearch release is downloaded. Alternatively we could document a docker run command that invokes the CLI, with users and users_roles mounted volumes. Personally (as a user) I think I would consider writing the content by hand to not bother with the extra tooling environment.

  2. ECK provides a higher-level abstraction for users, and roles keep the original format (described in this comment). End users provide one Kubernetes secret per user. The secret has its own ECK-specific format, but includes at least the user name, roles, and either the cleartext password or its hash. It may be convenient to have the cleartext password stored in that secret, so the secret can be mounted into other Kubernetes Pods that need to connect to Elasticsearch with the user credentials. ECK would internally derive those users into the right file realm format, compute password hashes if not provided, and merge this "external" file realm with the existing "internal" one.

Happy to hear your thoughts about it @jasontedor. There might be better alternatives we did not consider.

@anyasabo
Copy link
Contributor

anyasabo commented Feb 15, 2020

create the secret ourselves, which means username and roles must be specified in the CRD

Agreed this is challenging, especially because k8s does not provide any functionality to encrypt any resource types at rest, only secrets. I didn't even see any k/k issue where this was brought up as a proposal. We should avoid storing any secret data in anything but a secret resource. I looked into other operators that might need similar functionality, and saw that cert-manager stores the secret data for certs in secrets as well for reference.

maybe it's OK to not have the password generation feature.

This makes sense to me I think. It seems like this is an overall challenge in k8s and not just for us, and I don't see any good way to use it. Anyone who is determined to use autogenerated passwords is likely to have this need for more than just ECK and can use a broader solution.


I thought it might be okay to require the hashes be generated, since I figured most people using this feature would have automation around it and would not be handrolling the secrets. But it appears that Helm does not currently have a bcrypt function. They use sprig, which has bcrypt implemented, but only as part of an htpasswd function. We may be able to open a PR with bcrypt support though if we wanted to.

That said, as @sebgl if we can generate the hash at runtime and there is no need to update the secret with the hash, I think I am okay with making the hash optional.

One thing I might be missing though for both the secret and the volume array suggestions -- the assumption is that we would aggregate all secrets/volumes into one secret which is then mounted into the pod, correct? Is there a way to easily do that for volumes? Compared to how we do it currently

@jasontedor
Copy link
Member

The file format seems simple enough to not bother with invoking a separate CLI and JVM in ECK container, or in every Elasticsearch container.

The concern I have comes from the past when Cloud managed plugins on their own, rather than using the elasticsearch-plugin command. It was difficult for Cloud to maintain, and it tied our hands when we wanted to evolve the internal structure of plugins in minor releases (similar to the keystore). While the users and users_roles files are simple today, I don't want to end up in a similar situation.

However, it seems that our docs offer more guarantees and say this:

The users file stores all the users and their passwords. Each line in the file represents a single user entry consisting of the username and hashed and salted password.

While it is possible to modify the users files directly using any standard text editor, we strongly recommend using the elasticsearch-users tool to apply the required changes.

Similarly, we specify the format of the users_roles file.

The users_roles file stores the roles associated with the users.

Each row maps a role to a comma-separated list of all the users that are associated with that role.

So while we strongly encourage the use of the tools, it seems that we are willing to accept manually managing these files and offer some guarantees about their internal structure that we couldn't break in a minor release. Given this, I would be okay with not relying on the tool as long as @tvernum is too.

@tvernum
Copy link

tvernum commented Mar 2, 2020

CLI issues

I think you're fine creating the files by hand, as long as you're strict about following exactly what ES produces.

The sorts of caveats are the ones like that which @pebrc mentioned earlier about needing $2a$ passwords. We only support files that are the same as what ES would produce itself, not something that looks pretty similar on visual inspection, etc.

But we do support people who do what @sebgl mentioned in point (1) of an earlier comment. Administrators may use ES tooling to produce a file in 1 place and copy it to their cluster. And we do not presume that their central tooling used the exact same version of those tools as their cluster has. There are people who generated files manually at some point in time and then copied those files into Ansible/Salt/Puppet/etc and now use them to deploy new clusters with potentially newer ES versions. That's fine.

We also, for obvious reasons, support upgrading old clusters with existing files to newer ES versions without running file migration tools. That's not to say we would never require a migration tool, but we haven't so far, and we'd do our best to support BWC for a reasonable period of time before we required admins to upgrade the file format.

However, you do risk missing out on potential new features that we might add. Theoretically, purely as an example, we might add IP restrictions, or date/time restrictions to users and support including those restrictions in the users file. For BWC reasons we would continue to read files that had the old format without those restrictions, but the more you implement inside ECK the harder it will be for you or your users to make use of that feature if we were to build it.
However, simply wrapping our CLI wouldn't change that situation.

Realm vs Roles

I do want to raise a comment about "file realm" vs "file roles".

I think this has settled to a place where users and roles are being handled separately which is good.

This issue, and all the conversations I was aware of until now had talked about the "File realm", but roles.yml is not part of the file realm.
It is a mistaken mental model to treat file-based users (the file realm) and file based roles (a roles provider) as connected in any way other than their storage mechanism.

API based users can be assigned roles from roles.yml and file based users can be assigned roles from the API. And LDAP users must be assigned roles from one of those 2, because there is no concept of "LDAP roles" (groups, yes, but not roles).

As a further example of this lack of connection, file based roles (roles.yml) are available even if the file realm is not enabled.

When administrators fail to appreciate the distinction between realms (which control users) and role providers, they end up with incorrect mental models which lead them to incorrect conclusions (like thinking that disabling the file realm with turn off support forroles.yml, or not understanding why an API user can have a role that isn't visible in the API).

I would encourage you to do your best to treat users and roles as different things, so that people who use ECK don't get led into that incorrect understanding.

Kibana Roles

Kibana roles rely on application privileges stored in ES roles. For example if you want to provide a user with access to a specific Kibana space, but not to other spaces, you need Kibana-specific applications privileges.
Though those privileges are stored inside ES, ES doesn't know how to interpret them as anything more than a list of privileges that mean something in Kibana and we do not provide any documentation or support for manually creating them through either roles.yml or the ES roles API. If you wish to create roles with Kibana specific privileges, you need to use the API that Kibana provides.

Which is to say, that no matter what you do in ECK, the roles you manage will be limited to use of ES features & APIs, and not Kibana features unless you create them via the Kibana API.

I think that's an entirely reasonable limitation, but you should be aware of it up front.

@sebgl
Copy link
Contributor Author

sebgl commented Mar 2, 2020

@tvernum thanks, your point about realms vs. roles is very important and I think we (I, at least) were mistaken here. Based on what you said it does not make sense to have a roles section inside realms.file.

I'm thinking about having both roles and fileRealm be part of a high-level auth field covering both authentication and authorization (security would be another name candidate but we already have TLS stuff elsewhere in the spec):

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.6.0
  auth:
    fileRealm:
    - secretName: my-secret-name
    - secretName: my-secret-2
    roles:
    - secretName: my-roles-1
    - secretName: my-roles-2

I think this also gives us room for extensibility in the future if needed:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.6.0
  auth:
    # createElasticUser: false
    # saml:
    # openIdConnect:
    fileRealm:
    - secretName: my-secret-name
      # type: user  
    - secretName: my-secret-2
      # type: rawContent
    roles:
    - secretName: my-roles-1
    - secretName: my-roles-2
    # - inline:
    #     my_role1:
    #         cluster: [ 'all' ]
    #     my_role2:
    #         run_as: [ 'clicks_watcher_1' ]
    #         cluster: [ 'monitor' ]

I think there is real value in making it possible for users to specify username, password and roles (arguments taken by the elasticsearch-users CLI) through Kubernetes secrets directly, instead of coming up with the right file realm content themselves.

kind: Secret
apiVersion: v1
metadata:
  name: my-user-1
stringData:
  name: my-user-name
  roles: role1,role2,role3
  password: mypassword
  passwordHash: # optional, can be used instead of `password`

Regarding the CLI usage, it is definitely simpler for ECK to not use it and build the expected file content directly. Which means we have to commit to support future changes in the file realm spec (eg. the hypothetic IP restriction field you mentioned). As you pointed out, we would have to anyway if we were to invoke the CLI ourselves.

@sebgl
Copy link
Contributor Author

sebgl commented Mar 4, 2020

We discussed this again with the team in a Zoom meeting.

We agreed on starting with a solution as close as possible to the Elasticsearch official documentation. That is, rely on Kubernetes secrets containing the expected file realm data:

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 7.6.0
  auth:
    roles:
    - secretName: my-roles-1
    fileRealm:
    - secretName: my-filerealm-secret

---

# File realm in ES format (from the CLI or manually assembled)
kind: Secret
apiVersion: v1
metadata:
  name: my-filerealm-secret
stringData:
  users: |-
    rdeniro:$2a$10$BBJ/ILiyJ1eBTYoRKxkqbuDEdYECplvxnqQ47uiowE7yGqvCEgj9W
    alpacino:$2a$10$cNwHnElYiMYZ/T3K4PvzGeJ1KbpXZp2PfoQD.gfaVdImnHOwIuBKS
    jacknich:{PBKDF2}50000$z1CLJt0MEFjkIK5iEfgvfnA6xq7lF25uasspsTKSo5Q=$XxCVLbaKDimOdyWgLCLJiyoiWpA/XDMe/xtVgn1r5Sg=
  users_roles: |-
    admin:rdeniro
    power_user:alpacino,jacknich
    user:jacknich

---

# Roles in ES format
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-1
stringData:
  click_admins: |-
    run_as: [ 'clicks_watcher_1' ]
    cluster: [ 'monitor' ]
    indices:
    - names: [ 'events-*' ]
      privileges: [ 'read' ]
      field_security:
        grant: ['category', '@timestamp', 'message' ]
      query: '{"match": {"category": "click"}}'
  another_role: |-
    cluster: [ 'all' ]

We expect end users to come up with the right file realm secret content one way or another, for example by using the elasticsearch-users tool.

We also agreed on how this is maybe not the most convenient way to create users and passwords in a Kubernetes environment, hence keep the door open for extensibility. Some options listed in this issue could be implemented in the future:

  • having per-user Secrets with name, roles, and password fields that can be mounted into other Pods for convenience
  • adding an additional users_cleartext field in the secret content to let ECK hash passwords for convenience
    Since it's pretty hard to come up with the best approach right now, let's wait and see users needs.

I'm updating the first post accordingly.

@robertgates55
Copy link

Know this isn't the correct ticket to ask this on, but thought I'd see if anyone can help.

I'm running ECK 1.0.1 - is there any way to add users as secrets, as alluded to in this #728 (comment) message? Is anything additional required to make it work, or is there documentation anywhere about the spec required?

@sebgl
Copy link
Contributor Author

sebgl commented Mar 9, 2020

I think it's better to embed the entire roles.yml file content into a roles.yml field in the roles secrets. Otherwise it's pretty tricky to get the yaml indentation right, both for users and for us in the operator code:

# Roles in ES format
kind: Secret
apiVersion: v1
metadata:
  name: my-roles-1
stringData:
  roles.yml: |-
    click_admins:
      run_as: [ 'clicks_watcher_1' ]
      cluster: [ 'monitor' ]
      indices:
      - names: [ 'events-*' ]
        privileges: [ 'read' ]
        field_security:
          grant: ['category', '@timestamp', 'message' ]
        query: '{"match": {"category": "click"}}'
    another_role:
      cluster: [ 'all' ]

(instead of the example in the comment above).

@jsenon
Copy link

jsenon commented Mar 12, 2020

Hi,

I just want to share how we actually implement user creation in our stack. Actually we create our users through batch command that is not so acceptable:

DEPLOY_CONTAINER_IMAGE="docker.io/jsenon/toolingcontainer:latest"
[...]
create_kub_secret () {
  echo ""
  echo "  Creating Kub secret for user $1, in namespace $3..."
  uppercase_user=$(echo $1 | tr '[:lower:]' '[:upper:]')
  cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: elasticsearch-${1}-user-creds
  namespace: ${3}
type: Opaque
stringData:
  ${uppercase_user}_ELASTICSEARCH_USER: $1
  ${uppercase_user}_ELASTICSEARCH_PASSWORD: $2
EOF
}
user_name="txxx-log-reader"
echo ""
echo "-----------------------------------------------------"
echo "Creating user "$user_name
echo "-----------------------------------------------------"
echo "  Generating password..."
user_password=$(cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | head -c 32)
echo "  Creating User in ElasticSearch..."
kubectl run -it ${DEPLOY_JOB_NAME} --image $DEPLOY_CONTAINER_IMAGE --rm=true -n $CURL_RUNNER_NS --restart=Never -- \
curl -k -u $USER:$PASSWORD -X POST $HOST:$PORT"/_security/user/"$user_name -H 'Content-Type: application/json' -d'
{
  "password" : "'$user_password'",
  "roles" : [ "dev_read_only" ],
  "full_name" : "Task manager log reader account",
  "email" : "[email protected]",
  "metadata" : {}
}
' && create_kub_secret $user_name $user_password delair-dev
echo ""

echo "-----------------------------------------------------"
echo "Restarting elastic-exporter"
echo "-----------------------------------------------------"
echo ""; kubectl delete -n elasticsearch $(kubectl -n elasticsearch get pods -l "app=elastic-exporter" -o=name)

Our goals is to provision Kibana user as a self service. In order to do that we expect to have a CRD for a user creation:

apiVersion: elasticuser/v1
kind: User
metadata:
  name: user-alpacino
spec:
  login: alpacino
  password: XXX

Regards

@perezjasonr
Copy link

perezjasonr commented Feb 23, 2021

Hello quick clarification,

stringData: users: |- rdeniro:$2a$10$BBJ/ILiyJ1eBTYoRKxkqbuDEdYECplvxnqQ47uiowE7yGqvCEgj9W alpacino:$2a$10$cNwHnElYiMYZ/T3K4PvzGeJ1KbpXZp2PfoQD.gfaVdImnHOwIuBKS jacknich:{PBKDF2}50000$z1CLJt0MEFjkIK5iEfgvfnA6xq7lF25uasspsTKSo5Q=$XxCVLbaKDimOdyWgLCLJiyoiWpA/XDMe/xtVgn1r5Sg=

does it only take bcrypt, I tried sha256 w/ salt and it didn't seem to like it. it looked like this:

kind: Secret
apiVersion: v1
metadata:
  name: my-roles-secret
  namespace: elastic-system
stringData:
  roles.yml: |-
    logstash_writer:
      # run_as: [ 'clicks_watcher_1' ]
      # cluster: [ 'monitor' ]
      indices:
      - names: [ 'logstash-*' ]
      # probably dont need all of these
        privileges: [ 'create', 'create_index', 'index', 'write' ]
      #  field_security:
      #    grant: ['category', '@timestamp', 'message' ]
      #  query: '{"match": {"category": "click"}}'
---
kind: Secret
apiVersion: v1
metadata:
  name: my-filerealm-secret
  namespace: elastic-system
stringData:
  users: |-
    user_name:<a sha256 salted password>
  users_roles: |-
    logstash_writer:user_name

i did not see logstash_writer among the roles in kibana, did something go wrong?

@perezjasonr
Copy link

or more important question, does this have to come from the elasticsearch users helper tool, not sure if something proprietary was going on there.

@idanmo
Copy link
Collaborator

idanmo commented Feb 24, 2021

The recommendation is to use the elasticsearch-users tool (afaik nothing proprietary is going on there).
You can see a recent discussion about this here.

does it only take bcrypt, I tried sha256 w/ salt and it didn't seem to like it. it looked like this:

The default password hashing algorithm in Elasticsearch is bcrypt, so the password hash you specify has to be aligned with that. The hashing algorithm is configurable using the xpack.security.authc.password_hashing.algorithm setting but I think there would be some caveats setting it up properly with ECK.

For further usability questions please use Elastic's discussion forums.

@perezjasonr
Copy link

perezjasonr commented Feb 24, 2021

thank you ill check out the discussion, i tried bcrypt as well and its not showing up in kibana (the user, or the custom role). Not sure if file realm doesn't actually get registered in kibana or not but also i dont see any complaints in particular anywhere about the user/filerealm format etc...

[edit]

about the kibana part, nvm...just read this

You should also be aware that you cannot add or manage users in the file realm via the user APIs and you cannot add or manage them in Kibana on the Management / Security / Users page

ive continued the discussion in the discuss forums though about if i can just use the bcrypt lib myself and provide the value in the secrets so thank you.

@kristjankullerkann
Copy link

For others as it was a bit painful for myself.

A python example to generate password hash which seems to work:

#!/usr/bin/env python

import bcrypt

passwd = b'secret'

salt = bcrypt.gensalt(rounds=10, prefix=b"2a")
hashed = bcrypt.hashpw(passwd, salt)

print(hashed)

@vishalk663
Copy link

Few things to consider as part of this design are:

It's useful in certain scenarios to be able to access the password for a certain account by mounting that secret in another pod. If plaintext passwords are being considered, we should think about a key-value structure for the secret that does not hinder such usage.

There have been some questions about changing the passwords of file realm users after the cluster is created. Currently this can be done manually in a disruptive way with the following steps:

  • edit the [cluster]-es-*-user secret and replace the entry with the new base64 encoded password
  • delete the [cluster]-es-xpack-file-realm secret and wait for it to be recreated by the operator (the reconciliation logic only compares usernames and the not the password hashes to see if the secret needs to be updated)
  • restart the Elasticsearch nodes so that they pick up the new user file

It would be good to have a simpler and safer process for changing passwords as that is a reasonable thing that users might want to do on a fairly regular basis.

Perhaps its also worth thinking a little bit more about the request in #2408 and create a design that does not explicitly rely on secrets only. I have seen environments where ephemeral secrets from Vault are injected into pods by a sidecar as a file volume instead of a secret.

We followed the same process after changing the elastic superuser password

  1. update the secret in [cluster]-es-*-user secret
  2. delete the [cluster]-es-xpack-file-realm secret
  3. then restart elasticsearch pod one by one by deleting the pods

but still in elasticsearch logs we are getting below error

Authentication to realm file1 failed - Password authentication failed for elastic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>feature Adds or discusses adding a feature to the product v1.1.0
Projects
None yet
Development

Successfully merging a pull request may close this issue.