Vault abstraction for Galaxy

[nuwang]

This PR adds a vault abstraction for Galaxy with 3 backends - hashicorp, database and custos. It supercedes the work done in: #9876

Storing extra user preferences in the vault

Configuring file sources to use the vault

Programmatic Usage

Basic vault usage

from galaxy.security.vault import Vault, VaultFactory

vault = VaultFactory.from_app(app)
vault.write_secret("/my/super/secret", "hello secret")
print(vault.read_secret("/my/super/secret"))

User Level Vault

UserVaultWrapper creates a namespaced portion of the vault. Values are stored in the vault at: /{prefix}/users/{user_id}/{key}.

from galaxy.security.vault import UserVaultWrapper 

user_vault = UserVaultWrapper(app.vault, trans.user)
user_vault.write_secret("/my/super/secret", "hello secret")
print(user_vault.read_secret("/my/super/secret"))

Slides:

https://bit.ly/galaxy_vault

Tasks:

• Add basic vault abstraction
• Add support for hashicorp vault
• Add support for database backend with rotatable keys
• Add unit tests
• Add integration tests
• Add support for custos (thanks to some changes in custos that @isururanawaka has made. A few more are pending to make things even better)
• Add support for saving user pref password creds in vault?
• Add dedicated vault secret page to user preferences? No need anymore
• Validate secret key paths
• Refactor dependencies
• Conditional installation during unit tests
• Make sure file sources can use the vault
• Add support for listing secrets? Custos does not support this atm.
• Add new "secret" type to extra user prefs which is not sent to the client
• Use plugin system for vault backends? Not necessary
• Add documentation on vault configuration

How to test the changes?

(Select all options that apply)

• I've included appropriate automated tests.
• This is a refactoring of components with existing test coverage.
• Instructions for manual testing are as follows:
  1. [add testing steps and prerequisites here if you didn't write automated tests covering all your changes]

License

• I agree to license these contributions under Galaxy's current license.
• I agree to allow the Galaxy committers to license these and all my past contributions to the core galaxy codebase under the MIT license. If this condition is an issue, uncheck and just let us know why with an e-mail to [email protected].

[dannon]

@nuwang Fantastic! I had no idea you were working on this -- I see you used hvac here; I looked at that and was hoping it'd make this straightforward, and it looks like it does! Can you outline the extra bits required for 'Add support for custos (pending some changes in custos)'? And, would you want anyone else to help building out the UI bits?

[nuwang]

@dannon Custos currently requires single sign-on to access the vault api. It's being changed to add support for access through a service account. Isuru is working on this and it should be fairly straightforward to integrate afterwards. I'll take an initial crack at the UI bits just so I'm forced to work on the UI for a change, but may ask for help if I run into issues. I don't expect there to be a lot to do there, but famous last words.

[jdavcs]

@nuwang Would you mind adding a unit test for the new table mapping as well? Here's an example (the top of the file has some documentation on what's what) - it's just for the columns, since there are no relationships. It verifies the mapping is setup correctly (and not accidentally broken down the road), and it keeps it consistent with the other model. (I'll add all this to the docs on how to edit the model as soon as we move to alembic)

[mvdbeek]

Very cool, that's going to be very useful. I guess there's a client part coming in a followup that adds the rendering of secrets ?

[mvdbeek]

I guess there's a client part coming in a followup that adds the rendering of secrets ?

Not sure what happened there, the first time I tried a secret the entire extra preferences weren't rendering at all, but seems to work now. That's why I thought we still need some client work.

[mvdbeek]

Should we also stop shipping the password to the client at all ? This seems like a security and transparency issue, since the admins decide how the value is stored a user has no clue that maybe their password is safe on one server but not the other ?

[nuwang]

Thanks for the detailed review and test @mvdbeek, it's very helpful. I'll work on moving the api test into integration tests, and merging the test you added.

Regarding the password vs secret distinction for the client, I don't have a strong opinion on it. I suppose if we make all password fields "write only", the client would never be able to store and retrieve a secure secret, even if there was some future use case in which it needed to. On the other hand, it would make the defaults more secure yes.

I suppose we could also add a visual distinction in the client, maybe some text distinguishing between write only secrets and read-write secrets?

[afgane]

Works really nice. I am having trouble configuring the Custos backend because it keeps complaining about custos-sdk not being installed (even though it is) so perhaps once added into conditional deps that will be resolved...

[nuwang]

@mvdbeek Have made the suggested changes. Look ok to merge?