Plan refactor of secret and non-secret configuration data

[thekaveman]

The existing Django models capture both secret configuration items and non-secret configuration items. The vast majority of fields are non-secret and do not require additional controls.

For the secret configuration, values are currently maintained in Azure Key Vault and read into a new app service environment each time the app restarts (e.g. with a new deploy, or if an engineer restarts the app manually during debugging).

We want to continue to maintain the majority non-secret configuration data simply and directly via Django models and stored in the Django database.

We need a plan for how to incorporate the secret values in an environment where we do not re-create the database each time the app restarts, and thus would not (necessarily) read from Key Vault at app startup time.

Acceptance Criteria

• Secret configuration continues to be maintained in Azure Key Vault
• Non-secret configuration is moved out of Azure Key Vault and is managed directly in the Django admin interface
• Secret configuration is not visible within the Django admin interface (yet -- need roles/permissions etc.)
• Engineers with appropriate access can update secret values in Azure Key Vault and apply the changes to the existing app deployment, perhaps requiring an app restart, but without requiring a new deployment

Additional context

• Write product spec for Admin interface MVP (Use by Internal Eng) #1313
• Benefits app config field mapping (March 2023)

Activities

Running list of activities related to this effort

• Revisit and update the Benefits app config field mapping (March 2023) -- ✨ UPDATED tab ✨
• Determine how to programatically read secret values
• Determine how to refactor model classes with secrets

Implementation plan

• Implement a simple helper script in the spirit of this one in data-infra; use Azure's python libraries to read a value from a Key Vault by secret name. Ensure the Key Vault / environment (e.g. dev, test, prod) is not hardcoded; create a setting from an environment variable or similar.
• Create a custom Django field type that inherits from models.TextField. The default field implementation stores the name of the secret. An instance method like get_secret_value() uses the helper script to get the value of the named secret on-demand.
• Using the field mapping spreadsheet, update all secret fields to have this new field type. Update the field names to include _secret_name
• Add new properties to each model that use the helper script above and the _secret_name field value to get the actual value of the secret. These properties should be named the same as the original (pre-refactor) secret field (e.g. without the _secret_name postfix)

[thekaveman]

Maybe helpful here: https://learn.microsoft.com/en-us/python/api/overview/azure/keyvault-secrets-readme?view=azure-python&source=recommendations

[thekaveman]

Could follow this pattern / helper function get_secret_by_name: https://github.com/cal-itp/data-infra/blob/main/packages/calitp-data-infra/calitp_data_infra/auth.py

[thekaveman]

We talked through this plan at a recent Dev Workshop, and made some adjustments based on the conversation. Those are reflected in the steps above.