Vault abstraction for Galaxy

doc/source/admin/galaxy_options.rst

@@ -4809,5 +4809,13 @@
 :Default: ``plugins/welcome_page/new_user/static/topics/``
 :Type: str
 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``vault_config_file``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-
+:Description:
+    Vault configuration
+    The value of this option will be resolved with respect to
+    <config_dir>.
+:Default: ``vault_conf.yml``
+:Type: str

doc/source/admin/special_topics/index.rst

@@ -13,6 +13,7 @@ Special Topics
    gtn
    job_metrics
    webhooks
+   vault
    performance_tracking
    bug_reports
    gdpr_compliance

doc/source/admin/special_topics/vault.md

@@ -0,0 +1,120 @@
+# Storing secrets in the vault
+
+Galaxy can be configured to store secrets in an external vault, which is useful for secure handling and centralization of secrets management.
+In particular, information fields in the "Manage information" section of the user profile, such as a dropbox password, can be configured to be stored
+in an encrypted vault instead of the database. Vault keys are generally stored as key-value pairs in a hierarchical fashion, for example:
+`/galaxy/user/2/preferences/aws/client_secret`.
+
+## Vault backends
+
+There are currently 3 supported backends.
+
+| Backend     | Description                                                                                                                                                                                                                                                                                                                                   |
+|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hashicorp   | Hashicorp Vault is a secrets and encryption management system. https://www.vaultproject.io/                                                                                                                                                                                                                                                   |
+| custos      | Custos is an NSF-funded project, backed by open source software that provides science gateways such as Galaxy with single sign-on, group management, and management of secrets such as access keys and OAuth2 access tokens. Custos secrets managament is backed by Hashicorp's vault, but provides a convenient, always-on ReST API service. |
+| database    | The database backend stores secrets in an encrypted table in the Galaxy database itself. It is a convenient way to get started with a vault, and while it supports basic key rotation, we recommend using one of the above 2 options in production.                                                                                           |
+
+## Configuring Galaxy
+
+The first step to using a vault is to configure the `vault_config_file` setting in `galaxy.yml`. 
+
+```yaml
+galaxy:
+  # Vault config file
+  # The value of this option will be resolved with respect to
+  # <config_dir>.
+  vault_config_file: vault_conf.yml
+```
+
+## Configuring vault_conf.yml
+
+The vault_conf.yml file itself has two basic fields:
+```yaml
+type: hashicorp  # required
+path_prefix: /galaxy  # optional
+...
+```
+
+The `type` must be a valid backend type. The `path_prefix` property indicates the root path under which to store all vault keys. If multiple Galaxy instances are using the same Vault, a prefix can be used to identify the galaxy instance uniquely for example.
+If no path_prefix is provided, the prefix defaults to `/galaxy`.
+
+## Vault configuration for Hashicorp Vault
+
+```yaml
+type: hashicorp
+path_prefix: /my_galaxy_instance
+vault_address: http://localhost:8200
+vault_token: galaxy_test_token
+```
+
+## Vault configuration for Custos
+
+```yaml
+type: custos
+custos_host: service.staging.usecustos.org
+custos_port: 30170
+custos_client_id: custos-jeREDACTEDye-10000001
+custos_client_sec: OGREDACTEDBSUDHn
+```
+
+## Vault configuration for database
+
+```yaml
+type: database
+path_prefix: /galaxy
+# Encryption keys must be valid fernet keys
+# To generate a valid key:
+#
+# Use the ascii string value as a key
+# For more details, see: https://cryptography.io/en/latest/fernet/#
+encryption_keys:
+  - 5RrT94ji178vQwha7TAmEix7DojtsLlxVz8Ef17KWgg=
+  - iNdXd7tRjLnSqRHxuhqQ98GTLU8HUbd5_Xx38iF8nZ0=
+  - IK83IXhE4_7W7xCFEtD9op0BAs11pJqYN236Spppp7g=
+```
+
+Secrets stored in the database are encrypted using Fernet keys. Therefore, all listed encryption keys must be valid fernet keys. To generate a new
+Fernet key, use the following Python code:
+
+```python
+from cryptography.fernet import Fernet
+Fernet.generate_key()
+```
+which would result in a byte string such as `b'pZDP8_baVs3oWT4597HJWCysm49j-XELONQ-EdoU0DE='`. Copy the string value as the encryption key.
+
+If multiple encryption keys are defined, only the first key is used to encrypt secrets. The remaining keys are tried in turn during decryption. This is useful for key rotation.
+We recommend periodically generating a new fernet key and rotating old keys.
+
+## Configuring user preferences to use the vault
+
+The `user_preferences_extra_conf.yml` can be used to automatically route secrets to a vault. An example configuration follows:
+
+```yaml
+preferences:
+    googledrive:
+        description: Your Google Drive account
+        inputs:
+            - name: client_id
+              label: Client ID
+              type: text
+              required: True
+            - name: client_secret
+              label: Client Secret
+              type: secret
+              store: vault
+              required: True
+            - name: access_token
+              label: Access token
+              type: password
+              store: vault
+              required: True
+            - name: refresh_token
+              label: Refresh Token
+              type: secret
+              store: vault
+              required: True
+```
+
+Note the `store: vault` property, which results in the property being stored in the vault. Note also that if you use `type: password`, the secret is sent to the client front-end,
+but specifying `type: secret` would mean that the values cannot be retrieved by the client, only written to, providing an extra layer of security.

lib/galaxy/app.py

@@ -45,6 +45,10 @@
 )
 from galaxy.quota import get_quota_agent, QuotaAgent
 from galaxy.security.idencoding import IdEncodingHelper
+from galaxy.security.vault import (
+    Vault,
+    VaultFactory
+)
 from galaxy.tool_shed.galaxy_install.installed_repository_manager import InstalledRepositoryManager
 from galaxy.tool_shed.galaxy_install.update_repository_manager import UpdateRepositoryManager
 from galaxy.tool_util.deps.views import DependencyResolversView
@@ -183,6 +187,8 @@ def __init__(self, **kwargs):
         # ConfiguredFileSources
         self.file_sources = self._register_singleton(ConfiguredFileSources, ConfiguredFileSources.from_app_config(self.config))
 
+        self.vault = self._register_singleton(Vault, VaultFactory.from_app(self))
+
         # We need the datatype registry for running certain tasks that modify HDAs, and to build the registry we need
         # to setup the installed repositories ... this is not ideal
         self._configure_tool_config_files()

lib/galaxy/app_unittest_utils/galaxy_mock.py

@@ -182,6 +182,7 @@ def __init__(self, **kwargs):
         self.enable_tool_shed_check = False
         self.monitor_thread_join_timeout = 1
         self.integrated_tool_panel_config = None
+        self.vault_config_file = kwargs.get('vault_config_file')
 
     @property
     def config_dict(self):

lib/galaxy/config/sample/galaxy.yml.sample

@@ -2353,3 +2353,7 @@ galaxy:
   # is relative to galaxy/static
   #welcome_directory: plugins/welcome_page/new_user/static/topics/
 
+  # Vault config file
+  # The value of this option will be resolved with respect to
+  # <config_dir>.
+  vault_config_file: vault_conf.yml

lib/galaxy/dependencies/__init__.py

@@ -33,6 +33,7 @@ def __init__(self, config_file, config=None):
         self.container_interface_types = []
         self.job_rule_modules = []
         self.error_report_modules = []
+        self.vault_type = None
         if config is None:
             self.config = load_app_properties(config_file=self.config_file)
         else:
@@ -151,6 +152,17 @@ def collect_types(from_dict):
             file_sources_conf = []
         self.file_sources = [c.get('type', None) for c in file_sources_conf]
 
+        # Parse vault config
+        vault_conf_yml = self.config.get(
+            "vault_config_file",
+            join(dirname(self.config_file), 'vault_conf.yml'))
+        if exists(vault_conf_yml):
+            with open(vault_conf_yml) as f:
+                vault_conf = yaml.safe_load(f)
+        else:
+            vault_conf = {}
+        self.vault_type = vault_conf.get('type', '').lower()
+
     def get_conditional_requirements(self):
         crfile = join(dirname(__file__), 'conditional-requirements.txt')
         for req in pkg_resources.parse_requirements(open(crfile).readlines()):
@@ -261,6 +273,12 @@ def check_weasyprint(self):
         # See notes in ./conditional-requirements.txt for more information.
         return os.environ.get("GALAXY_DEPENDENCIES_INSTALL_WEASYPRINT") == "1"
 
+    def check_custos_sdk(self):
+        return 'custos' == self.vault_type
+
+    def check_hvac(self):
+        return 'hashicorp' == self.vault_type
+
 
 def optional(config_file=None):
     if not config_file:

lib/galaxy/files/__init__.py

@@ -263,6 +263,18 @@ def is_admin(self):
         """Whether this user is an administrator."""
         return self.trans.user_is_admin
 
+    @property
+    def user_vault(self):
+        """User vault namespace"""
+        user_vault = self.trans.user_vault
+        return user_vault or defaultdict(lambda: None)
+
+    @property
+    def app_vault(self):
+        """App vault namespace"""
+        vault = self.trans.app.vault
+        return vault or defaultdict(lambda: None)
+
 
 class DictFileSourcesUserContext:
 
@@ -296,3 +308,11 @@ def group_names(self):
     @property
     def is_admin(self):
         return self._kwd.get("is_admin")
+
+    @property
+    def user_vault(self):
+        return self._kwd.get("user_vault")
+
+    @property
+    def app_vault(self):
+        return self._kwd.get("app_vault")

lib/galaxy/managers/context.py

@@ -50,6 +50,7 @@
 )
 from galaxy.model.base import ModelMapping
 from galaxy.security.idencoding import IdEncodingHelper
+from galaxy.security.vault import UserVaultWrapper
 from galaxy.structured_app import MinimalManagerApp
 from galaxy.util import bunch
 
@@ -198,6 +199,11 @@ class ProvidesUserContext(ProvidesAppContext):
     def user(self):
         """Provide access to the user object."""
 
+    @property
+    def user_vault(self):
+        """Provide access to a user's personal vault."""
+        return UserVaultWrapper(self.app.vault, self.user)
+
     @property
     def anonymous(self) -> bool:
         return self.user is None

lib/galaxy/model/__init__.py

@@ -76,6 +76,7 @@
 from sqlalchemy.ext.orderinglist import ordering_list
 from sqlalchemy.orm import (
     aliased,
+    backref,
     column_property,
     deferred,
     joinedload,
@@ -8420,6 +8421,17 @@ class LibraryDatasetCollectionAnnotationAssociation(Base, RepresentById):
     user = relationship('User')
 
 
+class Vault(Base):
+    __tablename__ = 'vault'
+
+    key = Column(Text, primary_key=True)
+    parent_key = Column(Text, ForeignKey(key), index=True, nullable=True)
+    children = relationship('Vault', backref=backref('parent', remote_side=[key]))
+    value = Column(Text, nullable=True)
+    create_time = Column(DateTime, default=now)
+    update_time = Column(DateTime, default=now, onupdate=now)
+
+
 # Item rating classes.
 class ItemRatingAssociation(Base):
     __abstract__ = True

lib/galaxy/model/migrate/versions/0180_add_vault_table.py

@@ -0,0 +1,25 @@
+import datetime
+
+from sqlalchemy import Column, DateTime, ForeignKey, MetaData, Table, Text
+
+now = datetime.datetime.utcnow
+meta = MetaData()
+
+vault = Table(
+    'vault', meta,
+    Column('key', Text, primary_key=True),
+    Column('parent_key', Text, ForeignKey('vault.key'), index=True, nullable=True),
+    Column('value', Text, nullable=True),
+    Column("create_time", DateTime, default=now),
+    Column("update_time", DateTime, default=now, onupdate=now)
+)
+
+
+def upgrade(migrate_engine):
+    meta.bind = migrate_engine
+    vault.create()
+
+
+def downgrade(migrate_engine):
+    meta.bind = migrate_engine
+    vault.drop()