[Pitch to Production] Sidecar retrieves secrets and provides them to application

[micahlee]

POC Branches

• [WIP] Update demo app to showcase updated sidecar conjurdemos/pet-store-demo#14
• [WIP] Update sidecar to fetch secrets #24

Acceptance Criteria

• Existing sidecar behavior is maintained

  • Access token is available in the shared volume location
  • Sidecar run as init container and as a persistent sidecar

• Sidecar executes on a periodic basis to:

  • Read a secrets.yml file mounted as a volume to the sidecar container
  • Fetch the requested secrets from Conjur/DAP
  • Write the fetched secrets to a shared volume location
  • Write the fetched secrets to the Kubernetes Secrets Manager

• Fetching frequency is configurable

• Secret value is exactly the value stored in Conjur with no additional whitespace (e.g. newline) and no removed whitespace

  CI Test TBD

• Summon failure and reason for failure are clear in the sidecar logs, at least no less than what summon provides in the output.

  • Failures we might expect:
    • Can't connect to Conjur host
      • Network error
      • DNS error
    • Certificate verification failure
    • Unauthorized
    • Secret 404
    • Secret 403
    • Secrets.yml doesn't exist
    • Out of memory, out of disk space

• Volume mount secret is updated after Summon fetches within a known elapsed time.

• Risk of storing storing secrets in K8s secrets manager

  • Encryption of etcd is the responsibility of the K8s cluster operator.
  • Sidecar will attempt to delete secrets when the pod stops, but is not guaranteed.

• Sidecar works on Stock OSS Kubernetes and OpenShift (for all supported versions)

• Automated Unit Tests include:

  • K8s secrets API interactions

  • Secrets volume interactions

    • Overflowing the volume quota

  • Conjur Authentication (Which should already exists)

  • Summon interaction (Conjur Secret Retrieval)

    • Conjur Unavailable
    • Secret doesn't exist
    • Secret unauthorized

  • Secrets.yml via config map

  • Large secret data:

    • Test for how Conjur would handle this? (In cyberark/conjur)
    • Test for how DAP would handle this? (In conjurinc/appliance)
    • Test for how Summon would handle this? (In cyberark/summon)
    • Test for how Sidecar would handle out-of-memory/out-of-disk from K8s?

• Sidecar container contains only what is needed for execution

• The two sidecar responsibilities are clearly separated in code

• The documentation clearly explains:

  • Skeleton/process for a new app
  • The differences made to a deployment to add the sidecar (before and after)
  • Which persona is responsible for which processes
  • Using secrets from the shared volume

• Container logs are clear and informative

Open Questions

Open Questions

• Can you place a quota constraint on a memory volumes in k8s?

  • Can we place a quota on a Memory volume?
  • How does kubernetes handle exceeding the quota?
  • How large a request would Conjur support?
  • We should suggest a best practice starting quota and document that this is application specific and is subject to change if the secret data exceeds the default.

• What happens if the application container has a write lock on the secret file when the sidecar attempts to update it?

• Pre-stop hook for pod?

  • What is the scope of the secret in the k8s secret manager?
    • Can we prepend with pod name, for example?

• Is there an issue if we delete the secrets in the pre-stop hook?

  • What happens if the pre-stop doesn't succeed? Can we guarantee that the secrets are deleted?
  • https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
  • If we can't guarantee that the secrets is removed, how much do we want to attempt to remove it?

• Should the sidecar attempt to delete secret files upon termination?

  • Should this be configurable?

• Review documentation for what an application developer should expect from the secrets in the volume mount. with @yvonne

• If the call to summon fails (network/permissions)

  • Should the secrets in the volume be deleted, or should the stale secrets remain?

• Can we detect that the volume is a memory volume?

Follow Up Work

Ideas for follow on improvement work are:

• Adding security scanning to CI

• Risk of Appliance/OSS DOS load handling

• Can we encrypt the secrets at rest (a shared key between the sidecar and the app)

  • To prevent storing plain text material in a misconfigured persistent volume or unencrypted etcd.

• Bump to "Wow" for overall OpenShift Conjur developer experience

  • K8s-conjur-demo should be a one line deployment
    • Sidecar injector
    • Curl piped to bash pattern
    • Prompt for inputs

• Warn if using a volume type other than EmptyDir, memory backed

  • https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.14/#pod-v1-core

• How to create a K8s Role instead of a ClusterRole?

• Look into using a volume implementation for fetching secrets

  • e.g. https://kubernetes.io/docs/concepts/configuration/secret/#using-secrets-as-files-from-a-pod under Mounted Secrets are updated automatically.

• Make demo scripts configurable rather than deploy all demos at once.

• Demo scripts should be more self-documenting (e.g. --dry-run). See: --dry-run option conjurdemos/kubernetes-conjur-demo#66

[micahlee]

From @shaharglazner regarding:

Can you place a quota constraint on a memory volumes in k8s?

Individual secrets are limited to 1MiB in size. This is to discourage creation of very large secrets which would exhaust apiserver and kubelet memory. However, creation of many smaller secrets could also exhaust memory. More comprehensive limits on memory usage due to secrets is a planned feature.

(From: https://kubernetes.io/docs/concepts/configuration/secret/)

[micahlee]

From @shaharglazner regarding:

Can we encrypt the secrets at rest (a shared key between the sidecar and the app)

https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/

[jodyhuntatx]

Risk of storing storing secrets in K8s secrets manager

• Encryption of etcd is the responsibility of the K8s cluster operator.
• Sidecar will attempt to delete secrets when the pod stops, but is not guaranteed.

This is how most k8s developers would prefer we provide secrets to them. It would make the product easier to use - something it could definitely benefit from. :) However it goes against good security hygiene as mapping secrets to container env vars exposes them as plain text in tools like Weave Scope. If this route is taken, healthy caveats/warnings/disclaimers must be included.

It would be most awesome if there were a way to poke secrets into shell environment variables, as Aqua does.

[izgeri]

Roping in @doodlesbykumbi and @sgnn7. Since Secretless imports these libraries and we don't sufficiently test that use case here, I'd like to make sure any changes related to this issue are on the radar of the Secretless team.