feature request: create secret with every entry in secret manager key

[shraykay]

Would it be possible to add an extra section under External Secrets that will just import all key/values under the stored AWS key and add them to a Kubernetes secret?

I would like to avoid having to explicitly list out every key and property every time I update or add a new k/v to a AWS secret.

[silasbw]

We've been waffling on adding options for more "advanced" processing like this. Mostly because we don't understand the impact/utility of adding them (and we're worried about the longer term maintenance cost), and how general to make the advanced options so that they might scale to backends we add in the future.

Can you help us understand the impact for you? How many key, property objects do you have in an ExternalSecret? How many ExternalSecrets are you juggling?

[shraykay]

we have about 50 ExternalSecrets with ~15 key value pairs within each. We primarily use the envFrom feature in deployments to be able to source directly without having to specify each one by one.

[silasbw]

OK, thank you. It seems like this is worth considering supporting. We should think through how. e.g., single entry in the secretDescriptor.data array? some property on secretDescriptor that's mutually exclusive with secretDescriptor.data, or do we support some well-defined merge behavior? ... We'd welcome some concrete suggestions / brainstorming :)

[shraykay]

I think introducing another field would be great, similar to envFrom (especially for maintaining backwards). for "well defined merge behavior", anything that's manually specified in the data area should override anything that's in the envFrom area.

Thank you for responding so quickly!

[Flydiverny]

I think having a basic edition dataFrom would cover most common use cases? To make it slightly more advanced you could allow merge / overrides from data entries.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: postgres-credentials
secretDescriptor:
  backendType: secretsManager
  dataFrom: /development/postgres/credentials

secrets manager secret

{
    "Name": "/development/postgres/credentials",
    "SecretString":  "{\"username\":\"my-pg-user\", \"password\":\"my-super-secret-password\"}",
}

Resulting secret

apiVersion: v1
kind: Secret
metadata:
  name: postgres-credentials
type: Opaque
data:
  username: bXktcGctdXNlcg== # my-pg-user
  password: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk # my-super-secret-password

---

If one wants to go crazy on flexibility I guess you could do something like below, merging from top to bottom, ie last in list last to override.

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: all-them-secrets
secretDescriptor:
  backendType: secretsManager
  data:
  - dataFrom: /development/postgres/credentials
  - key: /development/auth/secret
    name: authSecret
  - key: /development/postgres/my-app/username
    name: username
  - dataFrom: /development/mysql/credentials

secrets manager secret

{
    "Name": "/development/postgres/credentials",
    "SecretString":  "{\"username\":\"my-pg-user\", \"password\":\"my-super-secret-password\"}",
}
{
    "Name": "/development/auth/secret",
    "SecretString":  "my-auth-secret",
 }
{
    "Name": "/development/postgres/my-app/username",
    "SecretString":  "my-app-username",
}
{
    "Name": "/development/mysql/credentials",
    "SecretString":  "{\"mysqlUsername\":\"my-mysql-user\", \"mysqlPassword\":\"my-super-secret-mysql-pw\"}",
}

Resulting secret

apiVersion: v1
kind: Secret
metadata:
  name: all-them-secrets
type: Opaque
data:
  username: bXktYXBwLXVzZXJuYW1l # my-app-username
  password: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk # my-super-secret-password
  authSecret: bXktYXV0aC1zZWNyZXQ=  # my-auth-secret
  mysqlUsername: bXktbXlzcWwtdXNlcg== # my-mysql-user
  mysqlPassword: bXktc3VwZXItc2VjcmV0LW15c3FsLXB3 # my-super-secret-mysql-pw

While this might be useful in some cases or simple-ish, it wouldn't work well if for example the mysql credentials secret also has keys named username instead of mysqlUsername, which seems more likely since its specifically a mysql secret. Should we then allow for renaming keys or prefixing them.

Prefix example

...
  - dataFrom: /development/postgres/credentials
    prefixNames: postgres

Map example

...
  - dataFrom: /development/postgres/credentials
    mapNames: 
    - property: username
      name: postgresUsername
    - property: password
      name: postgresPassword

To yield secret like:

...
data:
  postgresUsername: bXktcGctdXNlcg== # my-pg-user
  postgresPassword: bXktc3VwZXItc2VjcmV0LXBhc3N3b3Jk # my-super-secret-password
...

[Flydiverny]

I suppose the map example could potentially replace current functionallity where you would do:

  - data: /development/postgres/credentials
    property: username
    name: postgresUsername 
  - data: /development/postgres/credentials
    property: password
    name: postgresPassword

Maybe one could expand on current functionality to provide the following toolset:

Extract single JSON property from backend

  - data: /development/postgres/credentials
    property: password
    name: postgresPassword

Extract all JSON properties from backend

  - data: /development/postgres/credentials
    properties: all

Extract specific JSON properties from backend

  - data: /development/postgres/credentials
    properties:
    - property: username
      name: postgresUsername
    - property: password
      name: postgresPassword

Extract plain text from backend

  - data: /development/postgres/credentials
    name: postgresCredentials

It still leaves the scenario where you might want all the keys from two different backend values which might have the same property names. (prefix example in comment above)

[silasbw]

Hey, thanks for thinking this through @Flydiverny. I think I'd prefer the simpler approach to begin with and something that mirrors an existing pattern in kubernetes. Your first example, with some tweaks (envFrom is an array and specify a data array):

apiVersion: kubernetes-client.io/v1
kind: ExternalSecret
metadata:
  name: postgres-credentials
secretDescriptor:
  backendType: secretsManager
  dataFrom:
  - /development/postgres/credentials
  - /development/api/credentials
  data:
  - key: /development/auth/secret
    name: authSecret

Similar to how envFrom handles invalid environment variable names, creating the Secret would succeed even if the operations needed to implement a dataFrom failed. E.g., if the value in AWS SM wasn't an object and failed a JSON.parse, the controller would still create the Secret.

Values in data would override anything specified in dataFrom.

Thoughts?

[Flydiverny]

#196 was merged adding support as seen above in #94 (comment) !

[shraykay]

@Flydiverny thank you!