Add functionality to upload connector templates

[Kelsey-Ethyca]

Overview

We want to allow QA to test connectors from the connector builder and allow customers to create and upload their own connectors.

A very rough proof-of-concept can be found at origin/custom_connector_templates, this was just meant as an experiment but when this feature came up in planning it made sense to share it, even in its incomplete state.

Code Changes

• Modified the connector registry to store the actual configs, datasets, and other assets instead of just file paths. This was to standardize the contents of the registry and allow us to also read configs, and datasets from in the newly created custom_connector_template database table.
• Added the /connector_template/register endpoint to be able to upload a SaaS connector template (a zip file containing a config and dataset).
  • The parts of the connector template are stored to the database and added to the registry so that a connector can be instantiated from this newly registered template
• The SaaS connector template validation that occurs at server startup now also loads custom templates from the database
• Custom functions are now loaded during server startup and dynamically when a template is registered via the API.

Concerns

I (@galvana), stopped working on this POC once I started having concerns about keeping these templates up-to-date. A template is validated during registration so we can assert that it will work in the current version of Fides but this cannot be guaranteed as we develop the SaaS framework or refactor Fides in general

• the SaaS config spec can change
• the modules referenced in a custom override function can change location or functionality
• we might need to change the custom override interface

Brainstorming Ideas

I started thinking it might be best to keep custom connector templates in separate repos and structure those projects in a way that we can run continuous tests against new versions of Fides. A Fides instance could then reference these well-structured connector repos and import the templates that way.

We might also decide that we want to solve this challenge in a future iteration, any thoughts @NevilleS?

[Roger-Ethyca]

moving to done