Merge pull request #65 from CodeGrade/chore/deployment-docs

Deployment Documentation and Clean Up

DEPLOYMENT.md

@@ -0,0 +1,114 @@
+# Deploying Orca
+
+This guide serves to describe how to deploy Orca and its components in a **production** environment.
+
+## General Requirements
+
+The following tools must be present on the host machine running Orca (where 'x' means any minor/patch sub-version):
+* Docker v27.x.x
+* PostgreSQL v10.x.x
+
+Both the Orchestrator and the Worker(s) will utilize these tools when interacting with grading jobs and grader images.
+
+### The `orca-grader-base` Docker Image
+
+All grader images that Orca may create on demand are built from the `orca-grader-base` Docker image; it is necessary that this must exist before running the system.
+
+The Dockerfile for the base image is located in the `worker/` directory, and can be built by running the following command:
+
+```bash
+cd worker/
+docker build -t orca-grader-base -f images/orca-grader-base.Dockerfile .
+```
+
+In particular, if the Orchestrator and Worker are being run on separate machine instances, **please ensure the Orchestrator's machine has this image built by its local Docker service**.
+
+## The Orchestrator
+
+The Orchestrator expects to be run with Node version 21.0.0.
+
+### Environment Variables
+
+The following environment variables need to be set upon deploying the orchestrator:
+
+* `ENVIRONMENT` - Used for logging configuration and other conditionals; this should be set to `PRODUCTION`.
+* `POSTGRES_URL` - Used for connecting to and interacting with Orca's database; see [this Prisma page](https://www.prisma.io/dataguide/postgresql/short-guides/connection-uris) for the URL format.
+* `API_PORT` - The port on which the API should listen for incoming connections.
+
+### Installing Dependencies
+
+Dependencies can be installed and built from the top level of the `orchestrator/` directory.
+
+```bash
+yarn install
+yarn workspaces run build
+```
+
+From there, Prisma will be installed, and migrations for setting up the database can be executed by running the following:
+
+```bash
+cd packages/db
+npx prisma migrate deploy
+```
+
+**NOTE: This will not work if the POSTGRES_URL is not set.**
+
+This will create the schemas needed for grading jobs and grader image builds to be created.
+
+### Running the Orchestrator: A Tale of Two Cities
+
+The Orchestrator is made up of two components: the Web API and the Image Builder Service.
+
+There are two ways to run them: (1) as two separate terminal sessions _or_ (2) with a single command.
+
+Option (1):
+
+In two different terminal sessions, run the following:
+
+* Session A: `yarn server`
+* Session B: `yarn image-builder`
+
+
+Option (2):
+
+The node tool `concurrently` can be utilized to run the API and Build Service in one session; this has been aliased to this command:
+
+```bash
+yarn start
+```
+
+## The Worker
+
+The Worker expects to be run with Python version 3.10.6.
+
+### Environment Variables
+
+The following environment variables need to be set upon deploying a worker instance:
+
+* `ENVIRONMENT` - Used for certain conditional logic on the worker; this should be set to `PRODUCTION`.
+* `POSTGRES_URL` - Used for fetching jobs from the queue; this is the same URL for the running PostgreSQL database instance.
+* `ORCA_WEB_SERVER_HOST` - Needed for fetching grader images that don't exist on the worker's machine; this should be whatever the Orcherstrator's API URL is.
+
+### Installing Dependencies
+
+To install all of its dependencies, run the following commands:
+
+```bash
+python -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+The `venv` Python command ensures installed packages are local to the `worker/` directory instead of installing them system-wide.
+
+**NOTE:** unless actively running the Worker instance, the python virtual environment can be exited by typing `deactivate` and hitting ENTER in a given terminal session.
+
+### Running the Worker
+
+The worker can be run by simply executing the following:
+
+```bash
+source .venv/bin/activate
+python -m orca_grader
+```
+

README.md

@@ -4,38 +4,16 @@ Orca is a grading job ecosystem to be used in tandem with Bottlenose, a web appl
 
 | Directory   | Description                                                                                                      |
 | ----------- | ---------------------------------------------------------------------------------------------------------------- |
-| docs/       | Documentation, design specs, and diagrams to provide developers with the proper knowledge to contribute to Orca. |
-| grading-vm/ | Contains the logic for grading a submission with a given `GradingJob`.                                           |
+| docs/       | Documentation, design specs, and diagrams to provide developers with knowledge useful in contributing to Orca. |
+| worker/ | Contains the logic for grading a submission with a given `GradingJob`.                                           |
 | web-client/ | React application logic for allowing professors and admins to manage the grading queue.                          |
-| web-server/ | Express API/back end for queue management.                                                                       |
+| orchestrator/ | Web API and Build service for grader image and grading job management.                                                                       |
 
 ## Stack
 
 Orca expects the following tools with these versions:
 
-- Redis 7.0 (Grading Queue)
-- Node 20.11.0 (Web Server and Client)
-- Python 3.10.6 (Grading VM)
-
-## Local Development
-
-All local development is expected to be done in an environment running on **Ubuntu 20.04**.
-
-This can either be done with a machine directly running the OS or with a Windows machine running WSL2.
-
-For instructions about how to set up Ubuntu through WSL2 and Windows Terminal, [visit this page](https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-10#1-overview). For a more "aesthetic" and informative bash experience, developers can also set up [Starship](https://starship.rs/). Make sure to also add VSCode integration with the WSL2 terminal.
-
-Orca also uses Docker for local development and as a feature in the Grading VM. Docker should be on the host machine, either with Docker Desktop or Docker Community Edition installed in the shell.
-
-### TypeScript Tools
-
-Orca uses `yarn` as its package manager over `npm`. It also implements `eslint` and `prettier` across its TypeScript components.
-
-### VSCode Extensions
-
-The following VSCode extensions are expected to be installed for the best development experience:
-
-- **WSL** - allows direct access to code with Ubuntu (bash) terminal.
-- **Prettier** - used for enforcing code style and formatting files.
-- **ESLinter** - used for enforcing TypeScript code guidelines and error detection.
-- **Python** - used for IDE linting of Python code.
+- Docker 27.x.x
+- Postgres 10.x.x
+- Node 21.0.0 (Web Server and Client)
+- Python 3.10.6 (Worker)

worker/README.md

@@ -14,101 +14,26 @@ This software component is set up through the use of Python's virtual environmen
 2. To activate the virtual environment, run `source .venv/bin/activate`
 3. Set up local packages with `python -m pip install -r requirements.txt`.
 
-These steps, as well as the building of Docker images utilized in multiple aspects of testing the worker, are all contained in a single shell script. This can be run with:
-
-```
-$ ./init_dev.sh
-```
+When finished developing or running the worker, the user may return to their "normal shell" by executing `deactivate`.
 
 ## Running the Grader
 
-For standard execution, run the following commands:
+Given the PostgreSQL database is running, simply run
 
 ```
-$ docker compose up
 $ python -m orca_grader
 ```
 
-**BONUS: Populate Redis with Jobs**
-
-Useful for both stress testing and generally populating the Redis queue, the following script can be run to enqueue 1000 jobs:
-
-```
-$ python -m orca_grader.tests.stress_tests.burst_scenario
-```
-
 ### Checking Output
 
-To emulate sending the result of a grading job to a response URL, the worker module has a built-in _echo server_ provided by a basic Express API run in a docker container (this is included in the `docker-compose.yml` file). 
+To emulate sending the result of a grading job to a response URL, the worker module has a built-in _echo server_ provided by a basic Express API run in a docker container (this is included in the `docker-compose.yml` file).
 
 GETing the URL `http://localhost:9001/job-output` will yield which IDs can be queried for their results, and GETing the URL `http://localhost:9001/job-output/:id` will give the result of grading that specific job.
 
-### Using Redis
-
-The grading vm, when run normally (e.g., as it would be in production), has the machine talk to a Redis DB, where it fetches grading jobs and processes them one at a time.
-
-To emulate this behavior, devs can kick off a local Redis DB server instance by running the following:
-
-```
-$ docker run --rm -p 6379:6379 --name redis-server redis:7
-```
-
-### With Containers
-
-By default, Orca's grading vm will execute a grading job within a given container (provided through a `GradingJob`'s `container_sha` property).
-
-To ensure it connects to Redis correctly, we will need to connect the Redis instance to a custom Docker network. We can do this by running the following:
-
-```
-$ docker network create orca-testing
-$ docker network connect orca-testing redis-server
-```
-
-For debugging, devs can optionally specify a custom command to the grading container such that instead of running the grading job, it will simply execute that command instead. For example:
-
-```
-$ python -m orca_grader --custom-container-cmd "echo hello world!"
-```
-
-This is useful for running sanity checks, such as ensuring containers are being spun up properly and are able to be killed if they timeout.
-
-### Without Containers
-
-Users may optionally run the grader without spinning up a container by adding the `--no-container` flag to the python startup command. This will run the grading job on the local machine instead.
-
-### Running a Single, Local Job
-
-Instead of infinitely looping and checking the Redis queue for new jobs to pop off and handle, the grader can alternatively take in a grading job specified in a local .json file, execute it, and exit. This can be specified with the `--local-job /PATH/TO/JOB` flag.
-
-### Local File Server
-
-Code files in a grading job must always be downloaded via HTTP/HTTPS. Fortunately, we've added a docker container that can easily be built and used for this.
-
-First, all files to be used in local testing must be added under the `images/testing/simple-server/files` directory.
-
-Then, run the following commands from this directory:
-
-```
-$ docker build -t simple-server -f images/testing/simple-server/Dockerfile images/testing/simple-server
-$ docker run -p 9000:9000 --rm -d --network orca-testing --name simple-server
-```
-
-The first command builds the server image by copying all files in the given directory to the server such that it's able to deliver them over HTTP. The second command runs the server in the background, exposing port 9000 for machines to access.
-
-Note that whether the grading job is executed from the container or the local machine will affect how local file URLs should be written under the `files` key in a grading job:
-
-If running without the container, the URL should be "http://localhost:9000/files/PATH/TO/FILE".
-
-If running with th contianer, the URL should be "http://simple-server:9000/files/PATH/TO/FILE". This is because docker does not allow containers to connect to each other through localhost. We are able to change the host name here since both the file server and grading container are connected to the same docker network, `orca-testing`.
-
 ## Running Unit Tests
 
 The worker features multiple test suites to ensure robustness. All of these suites can be run by execution the command:
 
 ```
 $ python -m orca_grader.tests.runner
 ```
-
-## Exiting the Virtual Environment
-
-When finished with development, the user may return to their "normal shell" by executing `deactivate`.