Vault abstraction for Galaxy

doc/source/admin/galaxy_options.rst

@@ -4781,5 +4781,13 @@
 :Default: ``plugins/welcome_page/new_user/static/topics/``
 :Type: str
 
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+``vault_config_file``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-
+:Description:
+    Vault configuration
+    The value of this option will be resolved with respect to
+    <config_dir>.
+:Default: ``vault_conf.yml``
+:Type: str

doc/source/admin/special_topics/index.rst

@@ -13,6 +13,7 @@ Special Topics
    gtn
    job_metrics
    webhooks
+   vault
    performance_tracking
    bug_reports
    gdpr_compliance

doc/source/admin/special_topics/vault.md

@@ -0,0 +1,124 @@
+# Storing secrets in the vault
+
+Galaxy can be configured to store secrets in an external vault, which is useful for secure handling and centralization of secrets management.
+In particular, information fields in the "Manage information" section of the user profile, such as AWS credentials, can be configured to be stored
+in an encrypted vault instead of the database. Vault keys are generally stored as key-value pairs in a hierarchical fashion, for example:
+`/galaxy/user/2/preferences/aws/client_secret`.
+
+## Vault backends
+
+There are currently 3 supported backends.
+
+| Backend     | Description                                                                                                                                                                                                                                                                                                                                   |
+|-------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| hashicorp   | Hashicorp Vault is a secrets and encryption management system. https://www.vaultproject.io/                                                                                                                                                                                                                                                   |
+| custos      | Custos is an NSF-funded project, backed by open source software that provides science gateways such as Galaxy with single sign-on, group management, and management of secrets such as access keys and OAuth2 access tokens. Custos secrets management is backed by Hashicorp's vault, but provides a convenient, always-on ReST API service. |
+| database    | The database backend stores secrets in an encrypted table in the Galaxy database itself. It is a convenient way to get started with a vault, and while it supports basic key rotation, we recommend using one of the other options in production.                                                                                           |
+
+## Configuring Galaxy
+
+The first step to using a vault is to configure the `vault_config_file` setting in `galaxy.yml`. 
+
+```yaml
+galaxy:
+  # Vault config file
+  # The value of this option will be resolved with respect to
+  # <config_dir>.
+  vault_config_file: vault_conf.yml
+```
+
+## Configuring vault_conf.yml
+
+The vault_conf.yml file itself has two basic fields:
+```yaml
+type: hashicorp  # required
+path_prefix: /galaxy  # optional
+...
+```
+
+The `type` must be a valid backend type: `hashicorp`, `custos`, or `database`. At present, only a single vault backend
+is supported. The `path_prefix` property indicates  the root path under which to store all vault keys. If multiple
+Galaxy instances are using the same vault, a prefix can  be used to uniquely identify the Galaxy instance.
+If no path_prefix is provided, the prefix defaults to `/galaxy`.
+
+## Vault configuration for Hashicorp Vault
+
+```yaml
+type: hashicorp
+path_prefix: /my_galaxy_instance
+vault_address: http://localhost:8200
+vault_token: vault_application_token
+```
+
+## Vault configuration for Custos
+
+```yaml
+type: custos
+custos_host: service.staging.usecustos.org
+custos_port: 30170
+custos_client_id: custos-jeREDACTEDye-10000001
+custos_client_sec: OGREDACTEDBSUDHn
+```
+
+Obtaining the Custos client id and client secret requires first registering your Galaxy instance with Custos.
+Visit [usecustos.org](http://usecustos.org/) for more information.
+
+## Vault configuration for database
+
+```yaml
+type: database
+path_prefix: /galaxy
+# Encryption keys must be valid fernet keys
+# To generate a valid key:
+#
+# Use the ascii string value as a key
+# For more details, see: https://cryptography.io/en/latest/fernet/#
+encryption_keys:
+  - 5RrT94ji178vQwha7TAmEix7DojtsLlxVz8Ef17KWgg=
+  - iNdXd7tRjLnSqRHxuhqQ98GTLU8HUbd5_Xx38iF8nZ0=
+  - IK83IXhE4_7W7xCFEtD9op0BAs11pJqYN236Spppp7g=
+```
+
+Secrets stored in the database are encrypted using Fernet keys. Therefore, all listed encryption keys must be valid fernet keys. To generate a new
+Fernet key, use the following Python code:
+
+```python
+from cryptography.fernet import Fernet
+Fernet.generate_key().decode('utf-8')
+
+If multiple encryption keys are defined, only the first key is used to encrypt secrets. The remaining keys are tried in turn during decryption. This is useful for key rotation.
+We recommend periodically generating a new fernet key and rotating old keys. However, before removing an old key, make sure that any data encrypted using that old key is no longer
+present or the data will no longer be decryptable.
+
+## Configuring user preferences to use the vault
+
+The `user_preferences_extra_conf.yml` can be used to automatically route secrets to a vault. An example configuration follows:
+
+```yaml
+preferences:
+    googledrive:
+        description: Your Google Drive account
+        inputs:
+            - name: client_id
+              label: Client ID
+              type: text
+              required: True
+            - name: client_secret
+              label: Client Secret
+              type: secret
+              store: vault
+              required: True
+            - name: access_token
+              label: Access token
+              type: password
+              store: vault
+              required: True
+            - name: refresh_token
+              label: Refresh Token
+              type: secret
+              store: vault
+              required: True
+```
+
+Note the `store: vault` property, which results in the property being stored in the vault. Note also that if you use `type: password`, the secret is sent to the client front-end,
+but specifying `type: secret` would mean that the values cannot be retrieved by the client, only written to, providing an extra layer of security.

lib/galaxy/app.py

@@ -45,6 +45,10 @@
 )
 from galaxy.quota import get_quota_agent, QuotaAgent
 from galaxy.security.idencoding import IdEncodingHelper
+from galaxy.security.vault import (
+    Vault,
+    VaultFactory
+)
 from galaxy.tool_shed.galaxy_install.installed_repository_manager import InstalledRepositoryManager
 from galaxy.tool_shed.galaxy_install.update_repository_manager import UpdateRepositoryManager
 from galaxy.tool_util.deps.views import DependencyResolversView
@@ -207,6 +211,8 @@ def __init__(self, **kwargs):
         # ConfiguredFileSources
         self.file_sources = self._register_singleton(ConfiguredFileSources, ConfiguredFileSources.from_app_config(self.config))
 
+        self.vault = self._register_singleton(Vault, VaultFactory.from_app(self))
+
         # We need the datatype registry for running certain tasks that modify HDAs, and to build the registry we need
         # to setup the installed repositories ... this is not ideal
         self._configure_tool_config_files()

lib/galaxy/app_unittest_utils/galaxy_mock.py

@@ -188,6 +188,7 @@ def __init__(self, **kwargs):
         self.enable_tool_shed_check = False
         self.monitor_thread_join_timeout = 1
         self.integrated_tool_panel_config = None
+        self.vault_config_file = kwargs.get('vault_config_file')
 
     @property
     def config_dict(self):

lib/galaxy/config/sample/galaxy.yml.sample

@@ -2332,3 +2332,7 @@ galaxy:
   # is relative to galaxy/static
   #welcome_directory: plugins/welcome_page/new_user/static/topics/
 
+  # Vault config file
+  # The value of this option will be resolved with respect to
+  # <config_dir>.
+  vault_config_file: vault_conf.yml

lib/galaxy/dependencies/__init__.py

@@ -33,6 +33,7 @@ def __init__(self, config_file, config=None):
         self.container_interface_types = []
         self.job_rule_modules = []
         self.error_report_modules = []
+        self.vault_type = None
         if config is None:
             self.config = load_app_properties(config_file=self.config_file)
         else:
@@ -151,6 +152,17 @@ def collect_types(from_dict):
             file_sources_conf = []
         self.file_sources = [c.get('type', None) for c in file_sources_conf]
 
+        # Parse vault config
+        vault_conf_yml = self.config.get(
+            "vault_config_file",
+            join(dirname(self.config_file), 'vault_conf.yml'))
+        if exists(vault_conf_yml):
+            with open(vault_conf_yml) as f:
+                vault_conf = yaml.safe_load(f)
+        else:
+            vault_conf = {}
+        self.vault_type = vault_conf.get('type', '').lower()
+
     def get_conditional_requirements(self):
         crfile = join(dirname(__file__), 'conditional-requirements.txt')
         for req in pkg_resources.parse_requirements(open(crfile).readlines()):
@@ -273,6 +285,12 @@ def check_weasyprint(self):
         # See notes in ./conditional-requirements.txt for more information.
         return os.environ.get("GALAXY_DEPENDENCIES_INSTALL_WEASYPRINT") == "1"
 
+    def check_custos_sdk(self):
+        return 'custos' == self.vault_type
+
+    def check_hvac(self):
+        return 'hashicorp' == self.vault_type
+
 
 def optional(config_file=None):
     if not config_file:

lib/galaxy/dependencies/conditional-requirements.txt

@@ -25,6 +25,10 @@ fs-gcsfs # type: googlecloudstorage
 fs-onedatafs # type: onedata
 fs-basespace # type: basespace
 
+# Vault backend
+hvac
+custos-sdk
+
 # Chronos client
 chronos-python==1.2.1
 

lib/galaxy/files/__init__.py

@@ -282,6 +282,18 @@ def is_admin(self):
         """Whether this user is an administrator."""
         return self.trans.user_is_admin
 
+    @property
+    def user_vault(self):
+        """User vault namespace"""
+        user_vault = self.trans.user_vault
+        return user_vault or defaultdict(lambda: None)
+
+    @property
+    def app_vault(self):
+        """App vault namespace"""
+        vault = self.trans.app.vault
+        return vault or defaultdict(lambda: None)
+
 
 class DictFileSourcesUserContext(FileSourceDictifiable):
 
@@ -315,3 +327,11 @@ def group_names(self):
     @property
     def is_admin(self):
         return self._kwd.get("is_admin")
+
+    @property
+    def user_vault(self):
+        return self._kwd.get("user_vault")
+
+    @property
+    def app_vault(self):
+        return self._kwd.get("app_vault")

lib/galaxy/managers/context.py

@@ -49,6 +49,7 @@
 from galaxy.model.base import ModelMapping
 from galaxy.model.scoped_session import galaxy_scoped_session
 from galaxy.security.idencoding import IdEncodingHelper
+from galaxy.security.vault import UserVaultWrapper
 from galaxy.structured_app import MinimalManagerApp
 from galaxy.util import bunch
 
@@ -197,6 +198,11 @@ class ProvidesUserContext(ProvidesAppContext):
     def user(self):
         """Provide access to the user object."""
 
+    @property
+    def user_vault(self):
+        """Provide access to a user's personal vault."""
+        return UserVaultWrapper(self.app.vault, self.user)
+
     @property
     def anonymous(self) -> bool:
         return self.user is None

lib/galaxy/model/__init__.py

@@ -77,6 +77,7 @@
 from sqlalchemy.ext.orderinglist import ordering_list
 from sqlalchemy.orm import (
     aliased,
+    backref,
     column_property,
     deferred,
     joinedload,
@@ -8479,6 +8480,17 @@ class LibraryDatasetCollectionAnnotationAssociation(Base, RepresentById):
     user = relationship('User')
 
 
+class Vault(Base):
+    __tablename__ = 'vault'
+
+    key = Column(Text, primary_key=True)
+    parent_key = Column(Text, ForeignKey(key), index=True, nullable=True)
+    children = relationship('Vault', backref=backref('parent', remote_side=[key]))
+    value = Column(Text, nullable=True)
+    create_time = Column(DateTime, default=now)
+    update_time = Column(DateTime, default=now, onupdate=now)
+
+
 # Item rating classes.
 class ItemRatingAssociation(Base):
     __abstract__ = True

lib/galaxy/model/migrate/versions/0180_add_vault_table.py

@@ -0,0 +1,25 @@
+import datetime
+
+from sqlalchemy import Column, DateTime, ForeignKey, MetaData, Table, Text
+
+now = datetime.datetime.utcnow
+meta = MetaData()
+
+vault = Table(
+    'vault', meta,
+    Column('key', Text, primary_key=True),
+    Column('parent_key', Text, ForeignKey('vault.key'), index=True, nullable=True),
+    Column('value', Text, nullable=True),
+    Column("create_time", DateTime, default=now),
+    Column("update_time", DateTime, default=now, onupdate=now)
+)
+
+
+def upgrade(migrate_engine):
+    meta.bind = migrate_engine
+    vault.create()
+
+
+def downgrade(migrate_engine):
+    meta.bind = migrate_engine
+    vault.drop()