Extract embedded secrets and make them configurable

CHANGELOG.md

@@ -2,7 +2,15 @@
 
 All changes that impact users of this module are documented in this file, in the [Common Changelog](https://common-changelog.org) format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## Unreleased
+## Unreleased [minor]
+
+_Full changeset and discussions: [#30](https://github.com/OpenTermsArchive/deployment/pull/30)._
+
+> Development of this release was supported by the [French Ministry for Foreign Affairs](https://www.diplomatie.gouv.fr/fr/politique-etrangere-de-la-france/diplomatie-numerique/) through its ministerial [State Startups incubator](https://beta.gouv.fr/startups/open-terms-archive.html) under the aegis of the Ambassador for Digital Affairs.
+
+### Changed
+
+- **Breaking:** Extract embedded secrets and make them configurable; update your inventory to include secrets through configuration
 
 ## 0.0.16 - 2023-12-04
 

README.md

@@ -62,13 +62,19 @@ Available playbooks for the engine application:
 
 Available [variables](https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html) are listed below, along with default values:
 
-| Variable | Description | Default value |
-| --- | --- | --- |
-| `ota_engine_config_path` | Path to the engine config file related to the inventory file | `../config/production.json` |
-| `ota_engine_declarations_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the declarations repository to use | `main` |
-| `ota_engine_snapshots_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the snapshots repository to use | `main` |
-| `ota_engine_versions_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the versions repository to use | `main` |
-| `ota_engine_declarations_directory` | Path of the directory where the code will be deployed on the server | Value declared in the `name` key in the engine config file |
+| Variable | Description | Default value | Required |
+| --- | --- | --- | --- |
+| `ota_engine_github_bot_private_key` | SSH private key contents for GitHub user with privileges on snapshots and versions repositories | No default value | ✔︎ |
+| `ota_engine_smtp_password` | Password for the SMTP server used for sending error notifications by email | No default value | - |
+| `ota_engine_sendinblue_api_key` | SendInBlue API key used to send email notifications | No default value | - |
+| `ota_engine_github_token` | When defined, this authentication token enables GitHub issue creation on the declarations repository | No default value | - |
+| `ota_engine_config_path` | Path to the engine config file, relative to the inventory file | `../config/production.json` | - |
+| `ota_engine_declarations_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the declarations repository to use | `main` | - |
+| `ota_engine_snapshots_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the snapshots repository to use | `main` | - |
+| `ota_engine_versions_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the versions repository to use | `main` | - |
+| `ota_engine_declarations_directory` | Path of the directory where the code will be deployed on the server | Value declared in the `name` key in the engine config file | - |
+
+For encryption of sensitive configuration entries, please refer to the [dedicated section](#encrypt-sensitive-configuration-entries).
 
 These variables can be overriden in the inventory file, for example:
 
@@ -108,11 +114,14 @@ Available playbooks for the Federated API application:
 
 Available variables are listed below, along with default values:
 
-| Variable | Description | Default value |
-| --- | --- | --- |
-| `ota_federated_api_repo` | Repository URL of the federated API code | `https://github.com/OpenTermsArchive/federated-api.git` |
-| `ota_federated_api_directory` | Path of the directory where the code will be deployed on the server | `federated-api` |
-| `ota_federated_api_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the federated API repository to use | `main` |
+| Variable | Description | Default value | Required |
+| --- | --- | --- | --- |
+| `ota_federated_api_repo` | Repository URL of the federated API code | `https://github.com/OpenTermsArchive/federated-api.git` | - |
+| `ota_federated_api_directory` | Path of the directory where the code will be deployed on the server | `federated-api` | - |
+| `ota_federated_api_branch` | [Git branch or tag](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddeftree-ishatree-ishalsotreeish) of the federated API repository to use | `main` | - |
+| `ota_federated_api_smtp_password` | Password for the SMTP server used for sending errors notifications by email. | - | - |
+
+For encryption of sensitive configuration entries, please refer to the [dedicated section](#encrypt-sensitive-configuration-entries).
 
 These variables can be overridden in the inventory file, for example:
 
@@ -137,6 +146,63 @@ Available [tags](https://docs.ansible.com/ansible/latest/user_guide/playbooks_ta
 
 - - -
 
+## Encrypt sensitive configuration entries
+
+Certain configuration entries contain sensitive information that should be encrypted to ensure security. Ansible provides a convenient way to encrypt such strings using its built-in [vault feature](https://docs.ansible.com/ansible/2.9/user_guide/vault.html):
+
+```sh
+ansible-vault encrypt_string --name <sensitive-config-name> <sensitive-config-content>
+```
+
+For example, to encrypt the GitHub bot private key used by the engine to push updates:
+
+```sh
+ansible-vault encrypt_string --name 'ota_engine_github_bot_private_key' '-----BEGIN OPENSSH PRIVATE KEY-----
+b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn
+…
+UlcCkBZ5IkI0eNAAAAE25kcG50QE1CUC1OZHBudC5sYW4BAgMEBQYH
+-----END OPENSSH PRIVATE KEY-----
+'
+```
+
+The encrypted result will look like this:
+
+```sh
+ota_engine_github_bot_private_key: !vault |
+          $ANSIBLE_VAULT;1.1;AES256
+          62313438616266383732353634343736623532666365643364396464633732613966636235616261
+          3136656665316437613434323561613732373361306161640a306132316531356537373862363838
+          66363763613833373530633831653163303961376331393761366261633561656463626563383931
+          3361643836623239660a333134626139626465303234313366313433653261376437316231363834
+          32643261303534366333383131633430396366343631656363663965633964663331346231663166
+          3331316462356461373134303666613035393335333139613639
+```
+
+Then it can be used directly in the inventory file:
+
+```yml
+all:
+  hosts:
+    127.0.0.1:
+      ansible_user: debian
+      ota_engine_config_path: ./engine_config.json
+      ota_engine_declarations_branch: new-feature
+      ota_engine_github_bot_private_key: !vault |
+          $ANSIBLE_VAULT;1.1;AES256
+          62313438616266383732353634343736623532666365643364396464633732613966636235616261
+          3136656665316437613434323561613732373361306161640a306132316531356537373862363838
+          66363763613833373530633831653163303961376331393761366261633561656463626563383931
+          3361643836623239660a333134626139626465303234313366313433653261376437316231363834
+          32643261303534366333383131633430396366343631656363663965633964663331346231663166
+          3331316462356461373134303666613035393335333139613639
+```
+
+Repeat the process for each sensitive configuration entry that needs encryption.
+
+Please note that the data will be stored unencrypted on the deployment server.
+
+- - -
+
 ## Development
 
 ### Requirements
@@ -168,41 +234,31 @@ Then the code can be deployed to the running machine with all the options descri
 
 ### Test collection
 
-Test locally the changes to the collection before opening a pull request:
+Testing the Ansible collection locally is crucial to ensure that changes function properly before submitting them as a pull request.
+
+The testing environment is preconfigured for Open Terms Archive maintainers. For other contributors, the configuration file `tests/engine_config.json` needs to be updated to specify repositories where they have authorizations. Additionally, the `ota_engine_github_bot_private_key` value in the inventory file `tests/inventory.yml` should be updated.
 
-Remove all traces of previous tests to ensure that changes do not work by coincidence:
+Follow these instructions to test the collection in a local environment:
+
+- Ensure you have a clean testing environment to prevent interference from previous configurations:
 ```sh
 vagrant destroy
 vagrant up
 ```
 
-Start by applying changes on the virtual machine:
-
+- Apply the changes to the virtual machine:
 ```sh
 ansible-playbook ../playbooks/engine/all.yml
 ```
 
-Connect through SSH to the virtual machine and check that everything works as intended:
+- Connect to the virtual machine to verify that changes were applied successfully:
 ```sh
-vagrant ssh
-pm2 logs
-```
-
-### Vagrant quick reference
-
-#### Connect to the virtual machine
-
-```sh
-vagrant up
-vagrant ssh  # use "vagrant" as password
+vagrant ssh # use "vagrant" as password
 ```
 
-#### Start again with a clean virtual machine
-
+- Check that everything works as intended within the virtual machine. Depending on the nature of changes made, you can monitor logs or execute commands to validate functionality:
 ```sh
-vagrant halt  # stop machine
-vagrant destroy  # remove machine
-vagrant up
+pm2 logs
 ```
 
 ---

playbooks/engine/infrastructure.yml

@@ -20,7 +20,6 @@
         - ansible_distribution != 'Debian' or (ansible_distribution == 'Debian' and ansible_facts['architecture'] != 'aarch64')
 
   roles:
-    - role: infrastructure/git
     - role: infrastructure/node
     - role: infrastructure/chromium
     - role: infrastructure/nginx

playbooks/federated_api/infrastructure.yml

@@ -4,6 +4,5 @@
   become: true
 
   roles:
-    - role: infrastructure/git
     - role: infrastructure/node
     - role: infrastructure/nginx

roles/engine/tasks/database.yml

@@ -5,8 +5,7 @@
     dest: '{{ engine_database_directory }}'
     clone: false
     update: false
-    accept_hostkey: true
-    key_file: '/home/{{ ansible_user }}/.ssh/ota-bot-key'
+    key_file: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
   # the `before` property of the return value can tell us if the repository has been cloned already or not,
   # see <https://docs.ansible.com/ansible/latest/collections/ansible/builtin/git_module.html#return-values>
   register: existing_repository
@@ -16,8 +15,7 @@
     repo: '{{ engine_database_repository }}'
     version: '{{ engine_database_branch }}'
     dest: '{{ engine_database_directory }}'
-    accept_hostkey: true
-    key_file: '/home/{{ ansible_user }}/.ssh/ota-bot-key'
+    key_file: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
   when: existing_repository.before is defined and not existing_repository.before # if existing_repository.before is null, then the repository is new
 
 - name: Remove existing locks in {{ engine_database_name }}
@@ -26,16 +24,10 @@
     state: absent
 
 - name: Get latest data from {{ engine_database_repository }}
-  ansible.builtin.command:
-    cmd: git fetch origin
-    chdir: '{{ engine_database_directory }}'
-
-- name: Clean {{ engine_database_name }} local copy
-  ansible.builtin.command:
-    cmd: git reset --hard origin/{{ engine_database_branch }}
-    chdir: '{{ engine_database_directory }}'
-
-- name: Ensure {{ engine_database_name }} is on branch {{ engine_database_branch }}
-  ansible.builtin.command:
-    cmd: git checkout {{ engine_database_branch }}
-    chdir: '{{ engine_database_directory }}'
+  ansible.builtin.git:
+    repo: '{{ engine_database_repository }}'
+    version: '{{ engine_database_branch }}'
+    dest: '{{ engine_database_directory }}'
+    update: true
+    force: true # discard any modified files
+    key_file: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'

roles/engine/tasks/main.yml

@@ -1,11 +1,28 @@
+- name: Add GitHub Bot SSH key
+  ansible.builtin.copy:
+    content: '{{ ota_engine_github_bot_private_key }}'
+    dest: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
+    owner: '{{ ansible_user }}'
+    group: '{{ ansible_user }}'
+    mode: "600"
+
+- name: Add GitHub SSH key to known_hosts
+  ansible.builtin.known_hosts:
+    name: github.com
+    key: "{{ item }}"
+    path: ~/.ssh/known_hosts
+  with_items: # GitHub's SSH key fingerprints can be found here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints
+    - "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
+    - "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="
+    - "github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk="
+
 - name: Install services declarations
   ansible.builtin.git:
     repo: '{{ app_config.services.repository }}'
     dest: '/home/{{ ansible_user }}/{{ ota_engine_declarations_directory }}'
     version: '{{ ota_engine_declarations_branch }}'
     force: true
-    accept_hostkey: true
-    key_file: '/home/{{ ansible_user }}/.ssh/ota-bot-key'
+    key_file: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
     depth: 1
   tags:
     - update-declarations
@@ -18,9 +35,10 @@
     - update-declarations
 
 - name: Add .env file
-  ansible.builtin.copy:
+  ansible.builtin.template:
     src: .env
     dest: '/home/{{ ansible_user }}/{{ ota_engine_declarations_directory }}/.env'
+    force: true
     mode: "644"
 
 - name: Add pm2 config file

roles/engine/templates/.env

@@ -0,0 +1,11 @@
+{{ ansible_managed | comment }}
+
+{% if ota_engine_sendinblue_api_key is defined %}
+SENDINBLUE_API_KEY={{ ota_engine_sendinblue_api_key }}
+{% endif %}
+{% if ota_engine_smtp_password is defined %}
+SMTP_PASSWORD={{ ota_engine_smtp_password }}
+{% endif %}
+{% if ota_engine_github_token is defined %}
+GITHUB_TOKEN={{ ota_engine_github_token }}
+{% endif %}

roles/federated_api/tasks/main.yml

@@ -1,11 +1,28 @@
+- name: Add GitHub Bot SSH key
+  ansible.builtin.copy:
+    content: '{{ ota_engine_github_bot_private_key }}'
+    dest: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
+    owner: '{{ ansible_user }}'
+    group: '{{ ansible_user }}'
+    mode: "600"
+
+- name: Add GitHub SSH key to known_hosts
+  ansible.builtin.known_hosts:
+    name: github.com
+    key: "{{ item }}"
+    path: ~/.ssh/known_hosts
+  with_items: # GitHub's SSH key fingerprints can be found here: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/githubs-ssh-key-fingerprints
+    - "github.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOMqqnkVzrm0SdG6UOoqKLsabgH5C9okWi0dh2l9GKJl"
+    - "github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEmKSENjQEezOmxkZMy7opKgwFB9nkt5YRrYMjNuG5N87uRgg6CLrbo5wAdT/y6v0mKV0U2w0WZ2YB/++Tpockg="
+    - "github.com ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCj7ndNxQowgcQnjshcLrqPEiiphnt+VTTvDP6mHBL9j1aNUkY4Ue1gvwnGLVlOhGeYrnZaMgRK6+PKCUXaDbC7qtbW8gIkhL7aGCsOr/C56SJMy/BCZfxd1nWzAOxSDPgVsmerOBYfNqltV9/hWCqBywINIR+5dIg6JTJ72pcEpEjcYgXkE2YEFXV1JHnsKgbLWNlhScqb2UmyRkQyytRLtL+38TGxkxCflmO+5Z8CSSNY7GidjMIZ7Q4zMjA2n1nGrlTDkzwDCsw+wqFPGQA179cnfGWOWRVruj16z6XyvxvjJwbz0wQZ75XK5tKSb7FNyeIEs4TT4jk+S4dhPeAUC5y+bDYirYgM4GC7uEnztnZyaVWQ7B381AK4Qdrwt51ZqExKbQpTUNn+EjqoTwvqNj4kqx5QUCI0ThS/YkOxJCXmPUWZbhjpCg56i+2aB6CmK2JGhn57K5mj0MNdBXA4/WnwH6XoPWJzK5Nyu2zB3nAZp+S5hpQs+p1vN1/wsjk="
+
 - name: Clone federated API repository
   ansible.builtin.git:
     repo: '{{ ota_federated_api_repo }}'
     dest: '/home/{{ ansible_user }}/{{ ota_federated_api_directory }}'
     version: '{{ ota_federated_api_branch }}'
     force: true
-    accept_hostkey: true
-    key_file: '/home/{{ ansible_user }}/.ssh/ota-bot-key'
+    key_file: '/home/{{ ansible_user }}/.ssh/ota-github-bot-key'
     depth: 1
 
 - name: Read the production config
@@ -22,9 +39,10 @@
     chdir: '/home/{{ ansible_user }}/{{ ota_federated_api_directory }}'
 
 - name: Add .env file
-  ansible.builtin.copy:
+  ansible.builtin.template:
     src: .env
     dest: '/home/{{ ansible_user }}/{{ ota_federated_api_directory }}/.env'
+    force: true
     mode: "644"
 
 - name: Add pm2 config file

roles/federated_api/templates/.env

@@ -0,0 +1,5 @@
+{{ ansible_managed | comment }}
+
+{% if ota_federated_api_smtp_password is defined %}
+SMTP_PASSWORD={{ ota_federated_api_smtp_password }}
+{% endif %}