Java based cli for managing Zebedee collection encryption keys. If the Zebedee Keyring encryption key is ever exposed
or compremised immediate action is required minimise the risk
and window of opportunity for malicious parties. The dp-zebedee-collection-keyring-cli-java
cli rekey
command
automates the process of swapping the keyring encryption key with minimal fuss.
The rekey
command will:
- Create a
tar.gz
back up of the currentzebedee/keyring
dir in case a rollback is required. - Decrypt each key file using the current secret key.
- Re-encrypt each collection key with the new secret key and write the output back to the
/zebedee/keyring
dir. - Verify that all keys have been re-encrypted.
(These steps assume rekey
is being run on the prod environment).
-
Use the collection-keyring-secrets-generator tool to generate a new set of
SecretKey
andInitVector
values. This tool will output the values in the format required byrekey
.⚠️ These are now production secrets please ensure they are kept safe and secure and are not accidentally shared. :warning: -
Update the following Zebedee CMS secrets in dp-configs with the new values you generated in the step #1. Keep a copy of the original key/iv values as these will be required later.
"KEYRING_SECRET_KEY": "<new_key_value>", "KEYRING_INIT_VECTOR": "<new_iv_value>",
Don't merge the PR until the rekey command has run and the existing collection keys have been re-encrypted with the new key.
-
Stop Zebedee CMS via the Nomad UI and wait for the service to be confirmed as DEAD before continuing - we need to ensure that no new collections are created/updated while the rekey process is running.
-
ssh
on to the publishing box (Zebedee will be onpublishing_mount 1
) -
Create/start a new Java docker container with a volume mapped to the
zebedee_root
dir:sudo docker run -i -t --name rekey --userns=host -v /var/florence/zebedee/:/content openjdk:11 /bin/bash
Note: For local or other environments replace
/var/florence/zebedee/
with the appropriate zebedee root path. -
Install the pre-requisite tools in the container.
apt-get update && apt-get install maven && apt-get install git && apt-get install make
Clone the repo:
git clone https://github.com/ONSdigital/dp-zebedee-collection-key-cli-java.git
Build the cli jar:
cd dp-zebedee-collection-key-cli-java && make build
-
Run the
rekey
command updating the placeholders with the appropriate values. An explanation of each flag is listed in the table below:java -jar target/rekey.jar -k="<current_key>" -i="<current_iv>" -k2="<new_key>" -i2="<new_iv>" -z="<zebedee_root_dir>"
Flag Description -k
The current KEYRING_SECRET_KEY
value - required to decrypt the collection keys.-i
The current KEYRING_INIT_VECTOR
value - required to decrypt the collection keys.-k2
The new KEYRING_SECRET_KEY
value to encrypt the collection keys with (generated in step #1) .-i2
The new KEYRING_INIT_VECTOR
value - required to encrypt the collection keys (generated in step #1) .-z
The path to the Zebedee root directory - for dev/prod this will be /content
. -
If there are no errors and your output looks something like: Then
rekey
has completed successfully and all collections keys under$zebedee_root/keyring
should now be encrypted with the new key. -
Merge your dp-configs secrets PR.
-
Once the secrets pipeline has completed restart Zebedee CMS ensuring that the latest secrets have been picked up. To verify
rekey
has been successful:- Login to Florence and attempt to view content in any of the existing collections - If the collection keys have been successful mirgated to the new key you should be able to view the collection content without error.
-
Assuming everything has completed successfully - Exit the docker container and remove it:
sudo docker rm rekey
Congratuations - you have successfully completed your mission. 🚀 🎉
Before decrypting/re-encrypting the rekey
command will create a backup of the keyring directory -
keyring-backup-<timestamp>.tar.gz
. If rekey
is unsucessful or a rollback is required for any reason
-
Untar the backup tar.gz:
tar -xf keyring-backup-<timestamp>.tar.gz
-
Rename the untar output dir to
keyring
. This will replace the currentkeyring
directory with the original keyring dir containing collection keys encrypted with the old key.mv keyring-backup-<timestamp> keyring
-
Revert the dp-configs Zebedee CMS secrets (if required) and wait until the secrets pipeline has compeleted.
-
Restart Zebedee and check that the correct secrets have been picked up.
-
To verify the rollback has been successful:
- Login to Florence and attempt to view content in any of the existing collections - If the collection keys have been successful mirgated to the new key you should be able to view the collection content without error.