diff --git a/.github/workflows/docker.yaml b/.github/workflows/docker.yaml index 2df412c6f3..285e6ed585 100644 --- a/.github/workflows/docker.yaml +++ b/.github/workflows/docker.yaml @@ -27,7 +27,7 @@ jobs: run: pip install -r dev-requirements.txt - name: Build Fidesctl - run: nox -s build + run: nox -s "build(prod)" - name: Push Fidesctl run: nox -s push diff --git a/CHANGELOG.md b/CHANGELOG.md index a786b63bef..8bee10187b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -43,7 +43,6 @@ The types of changes are: * Okta, aws and database credentials can now come from `fidesctl.toml` config [#694](https://github.com/ethyca/fides/pull/694) * New `validate` endpoint to test aws and okta credentials [#722](https://github.com/ethyca/fides/pull/722) - ### Changed * Comparing server and CLI versions ignores `.dirty` only differences, and is quiet on success when running general CLI commands @@ -59,7 +58,7 @@ The types of changes are: * Replaced `make` with `nox` * Removed usage of `fideslang` module in favor of new [external package](https://github.com/ethyca/fideslang) shared across projects -* Added starting up the frontend server to `nox` +* Added a UI service to the docker-compose deployment [#757]() * `TestClient` defined in and shared across test modules via `conftest.py` [#759](https://github.com/ethyca/fides/pull/759) ### Docs diff --git a/README.md b/README.md index 6d05771220..8323bd97b0 100644 --- a/README.md +++ b/README.md @@ -9,38 +9,40 @@ [![Checked with mypy][mypy-image]][mypy-url] [![Twitter][twitter-image]][twitter-url] - ![Fidesctl banner](docs/fides/docs/img/fidesctl.png "Fidesctl banner") _[Join the waitlist](https://ethyca.com/waitlist/) to get started with our free, hosted version!_ ## :zap: Overview -Fides (*fee-dhez*, Latin: Fidēs) is an open-source tool that allows you to easily [declare your systems' privacy characteristics](https://ethyca.github.io/fides/tutorial/system/), [track privacy related changes](https://ethyca.github.io/fides/tutorial/policy/) to systems and data in version control, and [enforce policies](https://ethyca.github.io/fides/tutorial/pass/#evaluate-the-fidesctl-policies) in both your source code and [your runtime infrastructure](https://ethyca.github.io/fides/deployment/#step-5-install-fidesctl-cli-on-ci-build-server). +Fides (_fee-dhez_, Latin: Fidēs) is an open-source tool that allows you to easily [declare your systems' privacy characteristics](https://ethyca.github.io/fides/tutorial/system/), [track privacy related changes](https://ethyca.github.io/fides/tutorial/policy/) to systems and data in version control, and [enforce policies](https://ethyca.github.io/fides/tutorial/pass/#evaluate-the-fidesctl-policies) in both your source code and [your runtime infrastructure](https://ethyca.github.io/fides/deployment/#step-5-install-fidesctl-cli-on-ci-build-server). ![Fidesctl overview](docs/fides/docs/img/fidesctl-overview-diagram.png "Fidesctl overview") ## :rocket: Quick Start ### System Requirements -1. [Docker](https://www.docker.com/products/docker-desktop) (20.10.8+) and [Docker Compose](https://docs.docker.com/compose/install/) (1.29.0+) + +1. [Docker](https://www.docker.com/products/docker-desktop) (20.10.8+) and [Docker Compose](https://docs.docker.com/compose/install/) (1.29.0+) 2. [Python](https://www.python.org/downloads/) (3.8+) 3. [Nox](https://nox.thea.codes/en/stable/) (`pip install nox`) ### Getting Started -Fides is capable of generating a series of YAML configuration files to represent your stored data, processes, and organizations. These [configuration resources](https://ethyca.github.io/fides/language/resources/system.md) can then be exported into both a data map, and an Article 30-compliant Record of Processing Activities (RoPA). + +Fides is capable of generating a series of YAML configuration files to represent your stored data, processes, and organizations. These [configuration resources](https://ethyca.github.io/fides/language/resources/system.md) can then be exported into both a data map, and an Article 30-compliant Record of Processing Activities (RoPA). This guide will walk through generating a mock RoPA using predefined resources included in the [Fides repository](https://github.com/ethyca/fides). 1. Ensure `nox` and `docker` and installed locally, and clone the Fides repo. - + 2. From the root fides directory, run the following commands: ``` - nox -s cli + nox -s dev ``` - This will spin up the entire project and open a shell within the `fidesctl` container. - + + This will spin up the entire project and open a shell within the `fidesctl` container. + Once you see the `fides#` prompt (takes ~3 minutes the first time), you can run the next command: ``` @@ -49,7 +51,6 @@ This guide will walk through generating a mock RoPA using predefined resources i This builds the required images, spins up the database, and runs the initialization scripts. - 3. Use the `export datamap` command to generate a [data map](/docs/fides/docs/guides/generating_datamap.md) of the provided [demo resources](demo_resources/): ```sh @@ -61,10 +62,12 @@ This guide will walk through generating a mock RoPA using predefined resources i 4. View the newly-generated data map generated from the provided resources. - #### Controller +#### Controller + The header block at the top of the data map is composed of properties found in the [Organization resource](/demo_resources/demo_organization.yml). In a production deployment, this would be composed of publicly available information for your company or organization. - #### Article 30 Record of Processing Activities +#### Article 30 Record of Processing Activities + The remainder of the information on the data map is generated from the provided [configuration resources](https://ethyca.github.io/fides/language/resources/system.md). In a production environment, these could be [automatically generated](https://ethyca.github.io/fides/guides/generate_resources/) from your databases and system resources. The [Dataset resource](demo_resources/demo_dataset.yml) is primarily used to provide a list of categories of personal data, recorded here using the [Fides taxonomy](https://github.com/ethyca/fideslang), that your systems store or process, as well as their retention policies. Any Datasets referenced by a System will have this information included as rows of your data map. @@ -74,7 +77,7 @@ This guide will walk through generating a mock RoPA using predefined resources i Together, these configuration files build out an initial map of RoPA-required data and resources. 5. Assess the Organization and System datasets using the `--audit` flag. - + ``` fidesctl evaluate demo_resources/ --audit ``` @@ -126,10 +129,10 @@ The Fides core team is committed to providing a variety of documentation to help For more information on getting started with Fides, how to configure and set up Fides, and more about the Fides ecosystem of open source projects: -- Documentation: https://ethyca.github.io/fides/ -- Tutorial: https://ethyca.github.io/fides/tutorial/ -- Deployment: https://ethyca.github.io/fides/deployment/ -- Roadmap: https://github.com/ethyca/fides/projects +- Documentation: +- Tutorial: +- Deployment: +- Roadmap: - Website: www.ethyca.com/fides ### Support diff --git a/clients/admin-ui/.env.development b/clients/admin-ui/.env.development index 73f7d2bd1c..7d83fda2ad 100644 --- a/clients/admin-ui/.env.development +++ b/clients/admin-ui/.env.development @@ -1 +1,2 @@ -NEXT_PUBLIC_FIDESCTL_API=/api/v1 \ No newline at end of file +NEXT_PUBLIC_FIDESCTL_API=/api/v1 +NEXT_PUBLIC_FIDESCTL_API_SERVER=http://0.0.0.0:8080 diff --git a/clients/admin-ui/.env.production b/clients/admin-ui/.env.production index 73f7d2bd1c..7d83fda2ad 100644 --- a/clients/admin-ui/.env.production +++ b/clients/admin-ui/.env.production @@ -1 +1,2 @@ -NEXT_PUBLIC_FIDESCTL_API=/api/v1 \ No newline at end of file +NEXT_PUBLIC_FIDESCTL_API=/api/v1 +NEXT_PUBLIC_FIDESCTL_API_SERVER=http://0.0.0.0:8080 diff --git a/clients/admin-ui/README.md b/clients/admin-ui/README.md index 728f8b5ad0..8ffe870f3d 100644 --- a/clients/admin-ui/README.md +++ b/clients/admin-ui/README.md @@ -4,12 +4,8 @@ Admin UI for managing Fidesctl. ## Running Locally -1. Run `nox -s api` in top-level `fides` directory -2. In a new shell, `cd` into `clients/admin-ui`, install via `npm install`, then run `npm run dev`. - - - Alternatively, can run `nox -s admin_ui` to do both the install and run steps. - -3. Navigate to `http://localhost:3000/`. +1. Run `nox -s dev` in top-level `fides` directory +1. Navigate to `http://localhost:3000/`. ## Preparing for production diff --git a/clients/admin-ui/next.config.js b/clients/admin-ui/next.config.js index f0838dd42f..ef96539e18 100644 --- a/clients/admin-ui/next.config.js +++ b/clients/admin-ui/next.config.js @@ -26,11 +26,11 @@ const nextConfig = { return [ { source: `/api/v1/:path`, - destination: "http://0.0.0.0:8080/api/v1/:path/", + destination: `${process.env.NEXT_PUBLIC_FIDESCTL_API_SERVER}/api/v1/:path/`, }, { source: `/api/v1/:first/:second*`, - destination: "http://0.0.0.0:8080/api/v1/:first/:second*", + destination: `${process.env.NEXT_PUBLIC_FIDESCTL_API_SERVER}/api/v1/:first/:second*`, }, ]; }, diff --git a/docker-compose.yml b/docker-compose.yml index 40db2c1d52..e7a18efa14 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -3,7 +3,13 @@ services: image: ethyca/fidesctl:local command: uvicorn --host 0.0.0.0 --port 8080 --reload fidesapi.main:app healthcheck: - test: [ "CMD", "curl", "-f", "http://0.0.0.0:8080/api/v1/health" ] + test: + [ + "CMD", + "curl", + "-f", + "http://0.0.0.0:8080/api/v1/health" + ] interval: 15s timeout: 5s retries: 5 @@ -25,6 +31,23 @@ services: FIDESCTL__CLI__SERVER_PORT: "8080" FIDESCTL__API__DATABASE_HOST: "fidesctl-db" + fidesctl-ui: + image: ethyca/fidesctl:local-ui + command: npm run dev + depends_on: + - fidesctl + expose: + - 3000 + ports: + - "3000:3000" + volumes: + - type: bind + source: . + target: /fides + read_only: False + environment: + - NEXT_PUBLIC_FIDESCTL_API_SERVER=http://fidesctl:8080 + fidesctl-db: image: postgres:12 healthcheck: diff --git a/docs/fides/docs/guides/generating_datamap.md b/docs/fides/docs/guides/generating_datamap.md index 0fc2e998f0..3e9f83b884 100644 --- a/docs/fides/docs/guides/generating_datamap.md +++ b/docs/fides/docs/guides/generating_datamap.md @@ -5,7 +5,7 @@ Fides is capable of exporting a data map of your [resources](./../language/resou To follow along, ensure you have the Fides repository cloned and fidesctl installed. Additional support for running fidesctl locally can be found in the first step of the [Quick Start guide](https://github.com/ethyca/fides/#rocket-quick-start). ## Export the Demo Resources -First, ensure `fidesctl` is running with `nox -s cli`. +First, ensure `fidesctl` is running with `nox -s dev`. To apply and export the provided `demo_resources`, run the following commands: diff --git a/noxfiles/ci_nox.py b/noxfiles/ci_nox.py index cc50eb8939..8de2d8fbf4 100644 --- a/noxfiles/ci_nox.py +++ b/noxfiles/ci_nox.py @@ -12,21 +12,21 @@ START_APP, WITH_TEST_CONFIG, ) -from docker_nox import build_local_prod +from docker_nox import build from utils_nox import teardown RUN_STATIC_ANALYSIS = (*RUN_NO_DEPS, "nox", "-s") @nox.session() -def check_all(session: nox.Session) -> None: +def ci_checks(session: nox.Session) -> None: """ Runs all of the CI checks, except for 'pytest_external'. Excludes 'pytest_external' so that no additional secrets/tooling are required. """ teardown(session) - build_local_prod(session) + build(session, "test") black(session) isort(session) xenon(session) diff --git a/noxfiles/constants_nox.py b/noxfiles/constants_nox.py index d044bad4bf..d72aa03367 100644 --- a/noxfiles/constants_nox.py +++ b/noxfiles/constants_nox.py @@ -23,6 +23,7 @@ def get_current_tag() -> str: IMAGE_NAME = "fidesctl" IMAGE = f"{REGISTRY}/{IMAGE_NAME}" IMAGE_LOCAL = f"{IMAGE}:local" +IMAGE_LOCAL_UI = f"{IMAGE}:local-ui" IMAGE_LATEST = f"{IMAGE}:latest" # Disable TTY to perserve output within Github Actions logs @@ -44,4 +45,15 @@ def get_current_tag() -> str: CI_ARGS, IMAGE_NAME, ) -START_APP = ("docker-compose", "up", "-d", IMAGE_NAME) +START_APP = ("docker-compose", "up", "-d", "fidesctl") +START_APP_UI = ("docker-compose", "up", "-d", "fidesctl-ui") +START_APP_EXTERNAL = ( + "docker-compose", + "-f", + COMPOSE_FILE, + "-f", + INTEGRATION_COMPOSE_FILE, + "up", + "-d", + IMAGE_NAME, +) diff --git a/noxfiles/dev_nox.py b/noxfiles/dev_nox.py index 52de73e1e8..bc6b08ebdd 100644 --- a/noxfiles/dev_nox.py +++ b/noxfiles/dev_nox.py @@ -1,76 +1,18 @@ """Contains the nox sessions for running development environments.""" import nox -from constants_nox import ( - COMPOSE_FILE, - IMAGE_NAME, - INTEGRATION_COMPOSE_FILE, - RUN, - START_APP, -) -from docker_nox import build_local +from constants_nox import RUN, START_APP_EXTERNAL, START_APP_UI +from docker_nox import build @nox.session() -def reset_db(session: nox.Session) -> None: - """Reset the database.""" - build_local(session) +def dev(session: nox.Session) -> None: + """Spin up the entire application and open a development shell.""" + build(session, "dev") + build(session, "ui") session.notify("teardown") - session.run(*START_APP, external=True) - reset_db_command = (*RUN, "fidesctl", "db", "reset", "-y") - session.run(*reset_db_command, external=True) - - -@nox.session() -def api(session: nox.Session) -> None: - """Spin up the webserver.""" - build_local(session) - session.notify("teardown") - run_in_background = ("docker-compose", "up", IMAGE_NAME) - session.run(*run_in_background, external=True) - - -@nox.session() -def admin_ui(session: nox.Session) -> None: - """Spin up the frontend server in development mode""" - npm_install = ("npm", "install") - npm_run = ("npm", "run", "dev") - with session.chdir("clients/admin-ui"): - session.run(*npm_install, external=True) - session.run(*npm_run, external=True) - - -@nox.session() -def cli(session: nox.Session) -> None: - """Spin up a local development shell.""" - build_local(session) - session.notify("teardown") - session.run(*START_APP, external=True) - run_shell = (*RUN, "/bin/bash") - session.run(*run_shell, external=True) - - -@nox.session() -def cli_integration(session: nox.Session) -> None: - """Spin up a local development shell with integration images spun up.""" - build_local(session) - session.notify("teardown") - session.run( - "docker-compose", - "-f", - COMPOSE_FILE, - "-f", - INTEGRATION_COMPOSE_FILE, - "up", - "-d", - IMAGE_NAME, - external=True, - ) + if session.posargs == ["external"]: + session.run(*START_APP_EXTERNAL, external=True) + else: + session.run(*START_APP_UI, external=True) run_shell = (*RUN, "/bin/bash") session.run(*run_shell, external=True) - - -@nox.session() -def db_up(session: nox.Session) -> None: - """Spin up the application database in the background.""" - run_command = ("docker-compose", "up", "-d", "fidesctl-db") - session.run(*run_command, external=True) diff --git a/noxfiles/docker_nox.py b/noxfiles/docker_nox.py index 697156b346..4312499ff2 100644 --- a/noxfiles/docker_nox.py +++ b/noxfiles/docker_nox.py @@ -1,6 +1,12 @@ """Contains the nox sessions for docker-related tasks.""" import nox -from constants_nox import IMAGE, IMAGE_LATEST, IMAGE_LOCAL, get_current_tag +from constants_nox import ( + IMAGE, + IMAGE_LATEST, + IMAGE_LOCAL, + IMAGE_LOCAL_UI, + get_current_tag, +) def get_current_image() -> str: @@ -9,35 +15,39 @@ def get_current_image() -> str: @nox.session() -def build(session: nox.Session) -> None: +@nox.parametrize( + "image", + [ + nox.param("prod", id="prod"), + nox.param("dev", id="dev"), + nox.param("test", id="test"), + nox.param("ui", id="ui"), + ], +) +def build(session: nox.Session, image: str) -> None: """Build the Docker container for fidesctl.""" + + # The lambdas are a workaround to lazily evaluate get_current_image + # This allows the dev deployment to run without needing other dev requirements + build_matrix = { + "prod": {"tag": get_current_image, "target": "prod"}, + "dev": {"tag": lambda: IMAGE_LOCAL, "target": "dev"}, + "test": {"tag": lambda: IMAGE_LOCAL, "target": "prod"}, + "ui": {"tag": lambda: IMAGE_LOCAL_UI, "target": "frontend"}, + } + target = build_matrix[image]["target"] + tag = build_matrix[image]["tag"] session.run( "docker", "build", - "--target=prod", + f"--target={target}", "--tag", - get_current_image(), + tag(), ".", external=True, ) -@nox.session() -def build_local(session: nox.Session) -> None: - """Build the Docker container for fidesctl tagged 'local'.""" - session.run( - "docker", "build", "--target=dev", "--tag", IMAGE_LOCAL, ".", external=True - ) - - -@nox.session() -def build_local_prod(session: nox.Session) -> None: - """Build the Docker container for fidesctl tagged 'local' with the prod stage target.""" - session.run( - "docker", "build", "--target=prod", "--tag", IMAGE_LOCAL, ".", external=True - ) - - @nox.session() def push(session: nox.Session) -> None: """Push the fidesctl Docker image to Dockerhub.""" diff --git a/noxfiles/docs_nox.py b/noxfiles/docs_nox.py index 582d6940df..3226b10979 100644 --- a/noxfiles/docs_nox.py +++ b/noxfiles/docs_nox.py @@ -1,7 +1,7 @@ """Contains the nox sessions for developing docs.""" import nox from constants_nox import CI_ARGS, RUN -from docker_nox import build_local +from docker_nox import build @nox.session() @@ -12,7 +12,7 @@ def docs_build(session: nox.Session, build_type: str) -> None: """Build docs from the source code.""" session.notify("teardown") if build_type == "local": - build_local(session) + build(session, "dev") run_shell = (*RUN, "python", "generate_docs.py", "docs/fides/docs/") session.run(*run_shell, external=True)