Add the UI to the compose deployment

.github/workflows/docker.yaml

@@ -27,7 +27,7 @@ jobs:
         run: pip install -r dev-requirements.txt
 
       - name: Build Fidesctl
-        run: nox -s build
+        run: nox -s "build(prod)"
 
       - name: Push Fidesctl
         run: nox -s push

CHANGELOG.md

@@ -42,7 +42,6 @@ The types of changes are:
 * Okta, aws and database credentials can now come from `fidesctl.toml` config [#694](https://github.com/ethyca/fides/pull/694)
 * New `validate` endpoint to test aws and okta credentials [#722](https://github.com/ethyca/fides/pull/722)
 
-
 ### Changed
 
 * Comparing server and CLI versions ignores `.dirty` only differences, and is quiet on success when running general CLI commands
@@ -57,7 +56,7 @@ The types of changes are:
 
 * Replaced `make` with `nox`
 * Removed usage of `fideslang` module in favor of new [external package](https://github.com/ethyca/fideslang) shared across projects
-* Added starting up the frontend server to `nox`
+* Added a UI service to the docker-compose deployment [#757](<https://github.com/ethyca/fides/pull/757>)
 
 ### Docs
 

README.md

@@ -9,38 +9,40 @@
 [![Checked with mypy][mypy-image]][mypy-url]
 [![Twitter][twitter-image]][twitter-url]
 
-
 ![Fidesctl banner](docs/fides/docs/img/fidesctl.png "Fidesctl banner")
 
  _[Join the waitlist](https://ethyca.com/waitlist/) to get started with our free, hosted version!_
 
 ## :zap: Overview
 
-Fides (*fee-dhez*, Latin: Fidēs) is an open-source tool that allows you to easily [declare your systems' privacy characteristics](https://ethyca.github.io/fides/tutorial/system/), [track privacy related changes](https://ethyca.github.io/fides/tutorial/policy/) to systems and data in version control, and [enforce policies](https://ethyca.github.io/fides/tutorial/pass/#evaluate-the-fidesctl-policies) in both your source code and [your runtime infrastructure](https://ethyca.github.io/fides/deployment/#step-5-install-fidesctl-cli-on-ci-build-server).
+Fides (_fee-dhez_, Latin: Fidēs) is an open-source tool that allows you to easily [declare your systems' privacy characteristics](https://ethyca.github.io/fides/tutorial/system/), [track privacy related changes](https://ethyca.github.io/fides/tutorial/policy/) to systems and data in version control, and [enforce policies](https://ethyca.github.io/fides/tutorial/pass/#evaluate-the-fidesctl-policies) in both your source code and [your runtime infrastructure](https://ethyca.github.io/fides/deployment/#step-5-install-fidesctl-cli-on-ci-build-server).
 
 ![Fidesctl overview](docs/fides/docs/img/fidesctl-overview-diagram.png "Fidesctl overview")
 
 ## :rocket: Quick Start
 
 ### System Requirements
-1. [Docker](https://www.docker.com/products/docker-desktop) (20.10.8+) and [Docker Compose](https://docs.docker.com/compose/install/) (1.29.0+) 
+
+1. [Docker](https://www.docker.com/products/docker-desktop) (20.10.8+) and [Docker Compose](https://docs.docker.com/compose/install/) (1.29.0+)
 2. [Python](https://www.python.org/downloads/) (3.8+)
 3. [Nox](https://nox.thea.codes/en/stable/) (`pip install nox`)
 
 ### Getting Started
-Fides is capable of generating a series of YAML configuration files to represent your stored data, processes, and organizations. These [configuration resources](https://ethyca.github.io/fides/language/resources/system.md) can then be exported into both a data map, and an Article 30-compliant Record of Processing Activities (RoPA). 
+
+Fides is capable of generating a series of YAML configuration files to represent your stored data, processes, and organizations. These [configuration resources](https://ethyca.github.io/fides/language/resources/system.md) can then be exported into both a data map, and an Article 30-compliant Record of Processing Activities (RoPA).
 
 This guide will walk through generating a mock RoPA using predefined resources included in the [Fides repository](https://github.com/ethyca/fides).
 
 1. Ensure `nox` and `docker` and installed locally, and clone the Fides repo.
-   
+
 2. From the root fides directory, run the following commands:
 
     ```
-    nox -s cli
+    nox -s dev
     ```
-    This will spin up the entire project and open a shell within the `fidesctl` container. 
-
+
+    This will spin up the entire project and open a shell within the `fidesctl` container.
+
     Once you see the `fides#` prompt (takes ~3 minutes the first time), you can run the next command:
 
     ```
@@ -49,7 +51,6 @@ This guide will walk through generating a mock RoPA using predefined resources i
 
     This builds the required images, spins up the database, and runs the initialization scripts.
 
-
 3. Use the `export datamap` command to generate a [data map](/docs/fides/docs/guides/generating_datamap.md) of the provided [demo resources](demo_resources/):
 
     ```sh
@@ -61,10 +62,12 @@ This guide will walk through generating a mock RoPA using predefined resources i
 
 4. View the newly-generated data map generated from the provided resources.
 
-    #### Controller
+#### Controller
+
     The header block at the top of the data map is composed of properties found in the [Organization resource](/demo_resources/demo_organization.yml). In a production deployment, this would be composed of publicly available information for your company or organization.
 
-    #### Article 30 Record of Processing Activities 
+#### Article 30 Record of Processing Activities
+
     The remainder of the information on the data map is generated from the provided [configuration resources](https://ethyca.github.io/fides/language/resources/system.md). In a production environment, these could be [automatically generated](https://ethyca.github.io/fides/guides/generate_resources/) from your databases and system resources.
 
     The [Dataset resource](demo_resources/demo_dataset.yml) is primarily used to provide a list of categories of personal data, recorded here using the [Fides taxonomy](https://github.com/ethyca/fideslang), that your systems store or process, as well as their retention policies. Any Datasets referenced by a System will have this information included as rows of your data map.
@@ -74,7 +77,7 @@ This guide will walk through generating a mock RoPA using predefined resources i
     Together, these configuration files build out an initial map of RoPA-required data and resources.
 
 5. Assess the Organization and System datasets using the `--audit` flag.
-   
+
     ```
     fidesctl evaluate demo_resources/ --audit
     ```
@@ -126,10 +129,10 @@ The Fides core team is committed to providing a variety of documentation to help
 
 For more information on getting started with Fides, how to configure and set up Fides, and more about the Fides ecosystem of open source projects:
 
-- Documentation: https://ethyca.github.io/fides/
-- Tutorial: https://ethyca.github.io/fides/tutorial/
-- Deployment: https://ethyca.github.io/fides/deployment/
-- Roadmap: https://github.com/ethyca/fides/projects
+- Documentation: <https://ethyca.github.io/fides/>
+- Tutorial: <https://ethyca.github.io/fides/tutorial/>
+- Deployment: <https://ethyca.github.io/fides/deployment/>
+- Roadmap: <https://github.com/ethyca/fides/projects>
 - Website: www.ethyca.com/fides
 
 ### Support

clients/admin-ui/.env.development

@@ -1 +1,2 @@
-NEXT_PUBLIC_FIDESCTL_API=/api/v1
+NEXT_PUBLIC_FIDESCTL_API=/api/v1
+NEXT_PUBLIC_FIDESCTL_API_SERVER=http://0.0.0.0:8080

clients/admin-ui/.env.production

@@ -1 +1,2 @@
-NEXT_PUBLIC_FIDESCTL_API=/api/v1
+NEXT_PUBLIC_FIDESCTL_API=/api/v1
+NEXT_PUBLIC_FIDESCTL_API_SERVER=http://0.0.0.0:8080

clients/admin-ui/README.md

@@ -4,12 +4,8 @@ Admin UI for managing Fidesctl.
 
 ## Running Locally
 
-1. Run `nox -s api` in top-level `fides` directory
-2. In a new shell, `cd` into `clients/admin-ui`, install via `npm install`, then run `npm run dev`.
-
-   - Alternatively, can run `nox -s admin_ui` to do both the install and run steps.
-
-3. Navigate to `http://localhost:3000/`.
+1. Run `nox -s dev` in top-level `fides` directory
+1. Navigate to `http://localhost:3000/`.
 
 ## Preparing for production
 

clients/admin-ui/next.config.js

@@ -26,11 +26,11 @@ const nextConfig = {
     return [
       {
         source: `/api/v1/:path`,
-        destination: "http://0.0.0.0:8080/api/v1/:path/",
+        destination: `${process.env.NEXT_PUBLIC_FIDESCTL_API_SERVER}/api/v1/:path/`,
       },
       {
         source: `/api/v1/:first/:second*`,
-        destination: "http://0.0.0.0:8080/api/v1/:first/:second*",
+        destination: `${process.env.NEXT_PUBLIC_FIDESCTL_API_SERVER}/api/v1/:first/:second*`,
       },
     ];
   },

docker-compose.yml

@@ -3,7 +3,13 @@ services:
     image: ethyca/fidesctl:local
     command: uvicorn --host 0.0.0.0 --port 8080 --reload fidesapi.main:app
     healthcheck:
-      test: [ "CMD", "curl", "-f", "http://0.0.0.0:8080/api/v1/health" ]
+      test:
+        [
+          "CMD",
+          "curl",
+          "-f",
+          "http://0.0.0.0:8080/api/v1/health"
+        ]
       interval: 15s
       timeout: 5s
       retries: 5
@@ -25,6 +31,23 @@ services:
       FIDESCTL__CLI__SERVER_PORT: "8080"
       FIDESCTL__API__DATABASE_HOST: "fidesctl-db"
 
+  fidesctl-ui:
+    image: ethyca/fidesctl:local-ui
+    command: npm run dev
+    depends_on:
+      - fidesctl
+    expose:
+      - 3000
+    ports:
+      - "3000:3000"
+    volumes:
+      - type: bind
+        source: .
+        target: /fides
+        read_only: False
+    environment:
+      - NEXT_PUBLIC_FIDESCTL_API_SERVER=http://fidesctl:8080
+
   fidesctl-db:
     image: postgres:12
     healthcheck:

docs/fides/docs/guides/generating_datamap.md

@@ -5,7 +5,7 @@ Fides is capable of exporting a data map of your [resources](./../language/resou
 To follow along, ensure you have the Fides repository cloned and fidesctl installed. Additional support for running fidesctl locally can be found in the first step of the [Quick Start guide](https://github.com/ethyca/fides/#rocket-quick-start).
 ## Export the Demo Resources
 
-First, ensure `fidesctl` is running with `nox -s cli`.
+First, ensure `fidesctl` is running with `nox -s dev`.
 
 To apply and export the provided `demo_resources`, run the following commands:
 

noxfiles/ci_nox.py

@@ -12,21 +12,21 @@
     START_APP,
     WITH_TEST_CONFIG,
 )
-from docker_nox import build_local_prod
+from docker_nox import build
 from utils_nox import teardown
 
 RUN_STATIC_ANALYSIS = (*RUN_NO_DEPS, "nox", "-s")
 
 
 @nox.session()
-def check_all(session: nox.Session) -> None:
+def ci_checks(session: nox.Session) -> None:
     """
     Runs all of the CI checks, except for 'pytest_external'.
 
     Excludes 'pytest_external' so that no additional secrets/tooling are required.
     """
     teardown(session)
-    build_local_prod(session)
+    build(session, "test")
     black(session)
     isort(session)
     xenon(session)

noxfiles/constants_nox.py

@@ -23,6 +23,7 @@ def get_current_tag() -> str:
 IMAGE_NAME = "fidesctl"
 IMAGE = f"{REGISTRY}/{IMAGE_NAME}"
 IMAGE_LOCAL = f"{IMAGE}:local"
+IMAGE_LOCAL_UI = f"{IMAGE}:local-ui"
 IMAGE_LATEST = f"{IMAGE}:latest"
 
 # Disable TTY to perserve output within Github Actions logs
@@ -44,4 +45,15 @@ def get_current_tag() -> str:
     CI_ARGS,
     IMAGE_NAME,
 )
-START_APP = ("docker-compose", "up", "-d", IMAGE_NAME)
+START_APP = ("docker-compose", "up", "-d", "fidesctl")
+START_APP_UI = ("docker-compose", "up", "-d", "fidesctl-ui")
+START_APP_EXTERNAL = (
+    "docker-compose",
+    "-f",
+    COMPOSE_FILE,
+    "-f",
+    INTEGRATION_COMPOSE_FILE,
+    "up",
+    "-d",
+    IMAGE_NAME,
+)

noxfiles/dev_nox.py

@@ -1,76 +1,18 @@
 """Contains the nox sessions for running development environments."""
 import nox
-from constants_nox import (
-    COMPOSE_FILE,
-    IMAGE_NAME,
-    INTEGRATION_COMPOSE_FILE,
-    RUN,
-    START_APP,
-)
-from docker_nox import build_local
+from constants_nox import RUN, START_APP_EXTERNAL, START_APP_UI
+from docker_nox import build
 
 
 @nox.session()
-def reset_db(session: nox.Session) -> None:
-    """Reset the database."""
-    build_local(session)
+def dev(session: nox.Session) -> None:
+    """Spin up the entire application and open a development shell."""
+    build(session, "dev")
+    build(session, "ui")
     session.notify("teardown")
-    session.run(*START_APP, external=True)
-    reset_db_command = (*RUN, "fidesctl", "db", "reset", "-y")
-    session.run(*reset_db_command, external=True)
-
-
-@nox.session()
-def api(session: nox.Session) -> None:
-    """Spin up the webserver."""
-    build_local(session)
-    session.notify("teardown")
-    run_in_background = ("docker-compose", "up", IMAGE_NAME)
-    session.run(*run_in_background, external=True)
-
-
-@nox.session()
-def admin_ui(session: nox.Session) -> None:
-    """Spin up the frontend server in development mode"""
-    npm_install = ("npm", "install")
-    npm_run = ("npm", "run", "dev")
-    with session.chdir("clients/admin-ui"):
-        session.run(*npm_install, external=True)
-        session.run(*npm_run, external=True)
-
-
-@nox.session()
-def cli(session: nox.Session) -> None:
-    """Spin up a local development shell."""
-    build_local(session)
-    session.notify("teardown")
-    session.run(*START_APP, external=True)
-    run_shell = (*RUN, "/bin/bash")
-    session.run(*run_shell, external=True)
-
-
-@nox.session()
-def cli_integration(session: nox.Session) -> None:
-    """Spin up a local development shell with integration images spun up."""
-    build_local(session)
-    session.notify("teardown")
-    session.run(
-        "docker-compose",
-        "-f",
-        COMPOSE_FILE,
-        "-f",
-        INTEGRATION_COMPOSE_FILE,
-        "up",
-        "-d",
-        IMAGE_NAME,
-        external=True,
-    )
+    if session.posargs == ["external"]:
+        session.run(*START_APP_EXTERNAL, external=True)
+    else:
+        session.run(*START_APP_UI, external=True)
     run_shell = (*RUN, "/bin/bash")
     session.run(*run_shell, external=True)
-
-
-@nox.session()
-def db_up(session: nox.Session) -> None:
-    """Spin up the application database in the background."""
-    run_command = ("docker-compose", "up", "-d", "fidesctl-db")
-    session.run(*run_command, external=True)