podman should support native secrets

[kallisti5]

docker has the 'secret' command to manage local secrets, podman should offer the same.

• Selectively exposing secrets to the container in question
  • "username=ewrwer32"

Ideally, given kubernetes popularity, podman should offer some/all of the same "in-container" expose options k8s can.

• Environment Variable
• Mount point

Reference:

• https://kubernetes.io/docs/concepts/configuration/secret/

Extras:

• podman generate kube could generate k8s secret files as well 🔥

The current mount.conf method exposes the same secrets to all containers which is insecure. (lacks isolation)

[mheon]

From what I've seen, Docker secrets are mostly intended for use with Swarm, but I don't see much reason we can't use them with podman run. We'd need four commands for parity (secret create, secret ls, secret inspect, secret rm) plus a flag for podman create and podman run to add them.

[kallisti5]

It makes sense for "small single-node deployments" I mean, once you're up to things like k8s, you can graduate to their secret storage. This is nice because it (hopefully) avoids the "having clear text passwords on disk"

[rhatdan]

@kallisti5 Would you be interested in opening PRs to do this work? We would welcome it.

[kallisti5]

sure. I've been searching for secret storage solutions, and tbh the options outside of k8s kind of suck. Vault looks interesting, but Hasicorp is pretty hard to deal with if you don't have a contract with them.

I'll play around with some things and see if anything interesting comes of it. Would PR's 'going the right direction' that don't break the build be accepted, or is this a 'we need a working end-to-end solution' before accepting the PR?

[rhatdan]

Right direction would be fine.

[kallisti5]

ok. early work in progress here:

• https://github.com/kallisti5/libpod/commits/podman_secrets

Thinking of modeling secrets like volumes initially.

• Secrets using the 'local' driver get stored at:
  • secret_path="~/.local/share/containers/storage/secrets"
• A randomly generated encryption key stored in "~/.config/containers/libpod.conf" which encrypts all of the secrets on disk.
• Secrets presented to attached containers at /secrets/local/(name)

Having an encryption key in libpod.conf isn't great long-term, but I feel like it is a better solution than what people do today, and a good first iteration.

[rhatdan]

We already have a direcory /run/secrets in every container, better to place them there.
If this is a rootless case, should the secrets be encrypted with the a key key under ~/.ssh?

[rhatdan]

@mtrmac @vrothberg @mrunalp WDYT?

[mtrmac]

• It would be very nice (if not an outright requirement) if this could seamlessly graduate to k8s secrets (i.e. using the same concepts / formats, and ideally podman generate kube knowing how to export them in the right format). (OTOH personally I am very uninterested in Swarm compatibility, but others may have different opinions on that.)
• I am extremely skeptical about encrypting secrets and storing the key in a location with exactly the same permissions necessary to access the encrypted secrets. AFAICT all that can do is confuse casual users into thinking the data is somehow protected. Either be honest about the nonexistent confidentiality (i.e. just write the data to disk in plain text), or record the secrets and/or the secret-encrypting key in an actually more secure way (e.g. use a login-unlocked keyring, don’t record the key at all and force the user to enter a passphrase each time).

[rhatdan]

I agree on both points. We need this to be as integrated with the k8s work flow as possible. And at least for the rootless version, have the secrets encrypted with the user login session. At least then they would be secret when the user is not logged in.

No really good way to solve this for secrets used by root though, unless systemd has some magic thing.

[kallisti5]

Things like hashicorp vault require 3 of 5 unlock keys to decrypt the secret storage on startup.. unfortunately that doesn't make much sense in the non-daemon world. (And from a security standpoint still isn't super great since 'the app in the container still has to auth to vault'

So.. key things to solve:

• How to encrypt / decrypt secrets in a way that offers some real protection.
• How to export secrets to native k8s long-term as an upgrade path.
• Keeping compatibility. If podman run requires a password to unlock secrets, that isn't docker workflow compatible.

[rhatdan]

Would storing the unlock key in gnome-keyring or in kernel keyring be more in line with what the Docker Daemon does.
IE I specify the key one time and then it uses the one in the kernel keyring from then on.

[rhatdan]

@kallisti5 Any progress on this?

[kallisti5]

not at the moment. Too many projects, not enough man hours :-)
If anyone else wants to play with my changes / finish them / refactor them, feel free. (just post here so I know)

[mheon]

I've seen additional interest in this from #podman on Freenode, but no solid answers as to security model yet.

[github-actions]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[vrothberg]

That's something we can discuss next week. Would love to here @mtrmac's thoughts on it.

[rhatdan]

Added.

[Michael-Angel-Sec]

Thank you for your work on this feature. We are doing single node podman deployments and are looking forward to this feature.

[vrothberg]

@rhatdan, shall we reevaluate the priority of this request?

[rhatdan]

Yes I think this is something we should start work on.

[rhatdan]

@vrothberg do you have time to work on this? Otherwise I can add it to @QiWang19 List.

[mheon]

I think we still need a design conversation on how these should work.

[vrothberg]

I will be on PTO, but I will find time for that in March (along with the systemd work and auto updates). But anyone can pick it up :) There's also some performance improvements for c/image that I want to tackle in the near future.

I agree with Matt that we need a conversation on the design first; ideally a dedicated meeting where we can carve out what we want.

[xordspar0]

My usecase is similar to @dtitov 's. I have a single-node that I'm converting from docker-compose to podman. I already know Kubernetes, so I think having the same API as k8s would be awesome. The only gap for me is keeping the secrets out of my config, which is checked into git.

Ansible has support for local secrets. I use this for my (non-containerized) deployments today. The vault file is not checked into git, but I have a Makefile that prompts for secrets and creates the vault file if it's missing. So if I want to deploy from a new host, I can check out the git repo, run make deploy, copy-paste from my password manager, and deployments can be automated from then on. Maybe the design for how Podman stores secrets could learn something from that.

Come to think of it, maybe I can use Ansible to drive my Podman deployments. In that case I wouldn't need native secret support in Podman. However, if it would be really powerful if I could migrate to Kubernetes with no changes to my deployment spec.

[mheon]

The ansible secrets link is very interesting - I can see a model like that as being a viable way forward.

[mheon]

Ideally, we'd find a library to handle the cryptography/secret storage for us - I'd prefer not to have to write and maintain that ourselves.

[kallisti5]

Would storing the unlock key in gnome-keyring or in kernel keyring be more in line with what the Docker Daemon does.
IE I specify the key one time and then it uses the one in the kernel keyring from then on.

I think this is likely the best plan of action. Protecting secret storage with a local system keyring makes sense.

I would say the smartest thing would be to tie the support to Kubernetes.

I agree. Back in 2019 I was still using swarm, but k8s is easy enough to deploy now and widespread enough it's a better target.

Now, I converted my Docker Compose setup to podman setup following this article (using podman generate kube and podman play kube). In general, it works, but I realized that there's no support for secrets...

podman generate kube potentially spitting out kubernetes secrets from podman is pretty 🔥

Y'all would corner the market and make docker a thing of the past for desktop container development :-)

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@ashley-cui Have you thought about this at all?

[ashley-cui]

@rhatdan I have not, but would be happy to pick it up with some help

[rhatdan]

Sounds good, lets talk about this at watercooler.

[xordspar0]

Ideally, we'd find a library to handle the cryptography/secret storage for us - I'd prefer not to have to write and maintain that ourselves.

Ansible (written in Python) implements its Vaults with the symmetric encryption library cryptography: https://cryptography.io/en/latest/

I wonder if there's a similar symmetric encryption library for Go. That would be something to look into.

[xordspar0]

golang.org/x/crypto/nacl/secretbox is an option. Packages in the golang.org/x/ space are semi-official. Canonical JuJu apparently imports it. As a bonus, it is compatible with the NaCl secretbox library, which is written in C and has multiple bindings for other languages. The interface is only two functions: Open() and Seal(), which operate on []byte. I'm not sure how much we would need to implement from scratch to get the features we want, but this seems like a good start, or at least a candidate.

[mtrmac]

Careful with NaCl or any other format inventions (including things that compose FIPS-acceptable primitives), if we want this to be FIPS-certifiable. (Even deciding what crypto package to use requires having, and keeping up with, cryptography, sadly.)

IIRC the vague consensus from recent discussions has been:

• Use system-provided keyrings / facilities to store secrets.
• Or maybe just use an unencrypted plain-text file, explicitly not obfuscating anything, only providing a k8s-like feature to allow interoperating with k8s-targeted containers.
• Failing that, perhaps prompt interactively.

If I got it wrong, please correct me.

[mheon]

Initial prototype should be unencrypted. After that, we can look into encryption - best suggestion I heard was gnupg

[Radtoo]

pass/gopass and comparable tools could be great, just hand the password and other data from "pass show podman/container_id".

With a solution like it, you probably would have to mainly specify how you expect container secrets to be named so its clear for all other clients and podman which container they apply to, and whether you also support parsing additional YAML/TOML/JSON format data into additional runtime relevant variables if present. The rest is probably documentation.

Pass/gopass and such themselves are not hard to put in a container, and nothing should make it all that hard to plug in an entirely different tool as long as it can give podman a password and optionally YAML/TOML/JSON for other secret variables. I'm guessing many other tools also fairly trivially could be convinced to do this already, it wouldn't prejudice much if podman just expected to hand out an identifier and get strings in returns.

You might also really want to do this to avoid reinventing pinentry methods, on-disk encrypted storage formats with possible versioning, recipient management, n-factor authentication support with TOTP/HOTP/cryptokeys/smart cards/key and directory servers etc. and all the other things people came up with and needed before. The pass ecosystem and other tools have these covered.

[slewsys]

I am not familiar with k8s secrets, nor have I reviewed all the suggestions submitted above, so please forgive me if this has been covered: Mozilla sops. The idea is to store secrets in a JSON or YAML file whose keys remain unencrypted. For instance, a secrets file, secrets.yaml uploaded to a repository might look like:

db:
    user: ENC[AES256_GCM,data:BIlg,iv:3cUCeLA6a4AkZ7RhH2Z9cyLlzctr2ElOPEaE8SFmUSA=,tag:PWkoynFK+5DJs+eeo4bSNA==,type:str]
    password: ENC[AES256_GCM,data:rP0a0S1Kdg==,iv:xVOOIhPQVbIIc96tOKg/o4xk8gY+SGIxIX7MJ7w+fxQ=,tag:BrdIJg3GjcFGwNGWks19SQ==,type:str]
    #ENC[AES256_GCM,data:SmcmujwsKgfzG3bjiDp2uH1iNd567y37hI2XnJRseUjQZNetnGrjsIPX,iv:N7fGnW/n3ppY59d1JnnuMYK68z5xQ4PgGJAhklvnwFM=,tag:+0QVWzOLa87zLhynRf98Bg==,type:comment]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age:
    -   recipient: age1wv5pfqth5dgjvlr9rju7vafd3u8k33pslekuqls93v2n7a8445hqzrxqsp
        enc: !!binary |
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLU21oYjYrcE1ZSzUvNGIwYkVlbX
            NaVWZqbDdOOCtHYmdGOTlSQTRsZkRRCkc2azA0L09MOG13YTU2LzI1QU1kdFRtUElxZERV
            Y3BzWElCUmdmeWc2bEEKLS0tIEhPTDdLRS9mblZUYkNOc21nYXA3bzNIQ241RHFKN3pmV0
            JFQzN6bFlTdk0KC9ipdO2VGy2bP1zRXglV5+5OoQPTb5/pbwLaGQXV/kDNKj2eu631PafJ
            EE21Bocf5ATPmO8lxyRCSOft0nFFHg==
    lastmodified: '2020-11-14T10:26:01Z'
    mac: ENC[AES256_GCM,data:tJVIVvV2105zvgLBXToxblrqsEpQDAMaBUJLkOOAwAn5Mr/Wid9GwLYYOu9KAIz79fsThGlHerYhNM1f0HWglCom29r57PgKuQJc67hdlK3XX0nQ8Ph7ovVJ9ajlR46ZYOK8PXzvlUFIx6OUskjqSVMlCIX4kd6AdjJ9b154Uq8=,iv:mKQo5BQ1JiCvxMKl2K5jwnaKxfZuYYSMyWW9QA1Fiqo=,tag:7qqMyIBpS/VVwVH+rH04WQ==,type:str]
    pgp: []
    unencrypted_suffix: _unencrypted
    version: 3.6.1

but when edited, e.g., with the command such as:

sops --age age1wv5pfqth5dgjvlr9rju7vafd3u8k33pslekuqls93v2n7a8445hqzrxqsp secrets.yaml

would appear in an editor as:

db:
    user: eve
    password: c4r1b0u

sops supports encryption by various means, including PGP, GCP KMS, Azure Key Vault, etc. The authors recently made a good case for superseding PGP support with age, which is used in the example above.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ashley-cui]

Working on it, bot!!! slowly but surely

[vrothberg]

Note that it's not a hard requirement for 3.0. If we get it into a following minor release (e.g., 3.1) that's cool :)

[rhatdan]

This does not look like it made it, moving back to 3.1 it is under heavy development.

[ashley-cui]

closed via #9125