diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml
index 53d80da..8cd2cc0 100644
--- a/docs/mkdocs.yml
+++ b/docs/mkdocs.yml
@@ -27,12 +27,16 @@ markdown_extensions:
- attr_list
- admonition
- pymdownx.details
- - pymdownx.superfences
-
+ - pymdownx.superfences:
+ custom_fences:
+ - name: mermaid
+ class: mermaid
+ format: !!python/name:pymdownx.superfences.fence_code_format
nav:
- Home: 'index.md'
- 'Quick Start': 'quick-start.md'
- 'Deployment': 'deployment.md'
+ - 'Access Control': 'rbac.md'
- 'Development Guide':
- 'Local Development': 'dev-guide/local.md'
- 'Architecture': 'dev-guide/architecture.md'
diff --git a/docs/src/dev-guide/architecture.md b/docs/src/dev-guide/architecture.md
index 081b676..941d4e3 100644
--- a/docs/src/dev-guide/architecture.md
+++ b/docs/src/dev-guide/architecture.md
@@ -1,22 +1,115 @@
# Nesis Architecture
-The Nessus project is divided into three components.
-
-1. Frontend - This contains the ReactJS frontend and Node Express server (aka frontend-backend)
-2. API Backend - This is the backend API responsible for handling
- 1. Authentication
- 2. User management
- 3. Enforcing Role based access control
- 4. Process scheduling
- 5. And more
-6. RAG Engine - This is response for
- 7. Converting documents into embeddings
- 8. Interfacing with any OpenAI compatible endpoints
+The Nessus project is divided into four components.
+
+1. Frontend - This contains the ReactJS frontend and Node Express server (aka frontend-backend).
+2. API Backend - This is the backend API responsible for handling;
+ 1. Authentication.
+ 2. User management.
+ 3. Enforcing Role based access control.
+ 4. Process scheduling.
+ 5. And more.
+3. RAG Engine - This is response for;
+ 7. Converting documents into embeddings.
+ 8. Interfacing with any OpenAI compatible endpoints.
+4. Vector DB - For storing vector embeddings.
+
+
+## Sequence
+
+This is a simple sequence showing how requests flow when a user chats with their private documents.
+
+``` mermaid
+sequenceDiagram
+ autonumber
+ actor User
+ User->>Frontend: quesion
+ Frontend->>API: token, query
+ API->>API: validate token
+ API->>API: check CREATE action on prediction (rbac)
+ API->>API: collect datasources valid (READ) for user (rbac)
+ alt no datasources exists
+ API->>Frontend: 403
+ Frontend->>User: error message
+ else datasources exists:
+ API->>API: add to datasource collection
+ end
+ alt no CREATE prediction policy for user
+ API->>Frontend: 403
+ Frontend->>User: error message
+ end
+ API->>RAG: send request(context=True, filter=[datasources])
+ RAG->>VectorDB: query for document chunks
+ Note right of RAG: query *must* filter on 'datasource(s)'
+ VectorDB->>RAG: document chunks (filtered on datasource(s))
+ RAG->>RAG: create prompt with context (document chunks)
+ RAG->>LLM: prompt with chunks as context
+ LLM->>RAG: response
+ RAG->>API: response
+ API->>API: save response as prediction (remember CREATE on predictions)
+ API->>Frontend: response
+ Frontend->>User: success message
+```
+
+
+???+ note "Document Ingestion"
+
+ Documents are ingested by creating adding your document repository as a datasource and a manual
+ (one time only) or scheduled task.
+
## RAG Engine
+### Overview
+The RAG Engine is responsible for;
+
+1. Orchestrating the conversion of documents into embeddings using the ingestion process. Currently supported formats include pdf, mp3, mp4, tiff, png, jp[e]g, docx, pptx, epub.
+2. Persisting embeddings into vector store (See vector database section [below](#vector-database)).
+3. Receiving user queries (always) with context enabled and finding the necessary (filter=[datasource=?]) document chunks from the VectorDB.
+4. Formulating an OpenAI prompt with context from 3.
+5. Sending the formatted prompt to OpenAI.
+
### Vector Database
+A vector database component is critical to the functionality of the RAG engine. It stores vector embeddings and offers retrieval of
+document chucks from these embeddings.
+
+Nesis currently supports three vector databases;
+
+1. Postgres with pgvector extension. We have modified the bitnami postgres image and added the pgvector extension.
+2. Chromadb.
+3. Qdrant.
+
### Local Embeddings
+Nesis supports generating embeddings locally using HuggingFace embeddings models. The environment variable `NESIS_RAG_EMBEDDING_MODE` controls this.
+
+!!! warning "Huggingface vs OpenAI's Embeddings"
+
+ Huggingface (HF) embeddings use a dimension of 384 while OpenAI's default embeddings size varies.
+ The env variable `NESIS_RAG_EMBEDDING_DIMENSIONS` can be used to alter the dimention of embeddings
+ to suit your needs.
+
+ You may need to create a HF token in order to access any HF models. If you encounter an access denied during he ingestion process
+ set the `HF_TOKEN` with your HF token.
+
### OpenAI Embeddings
+
+Nesis can also use OpenAI to generate embeddings. By setting the environment variable `NESIS_RAG_EMBEDDING_MODE=openai`, Nesis would use the configured OpenAI
+endpoint to generate embeddings.
+
+!!! note
+
+ To use OpenAI embeddings, an `OPENAI_API_KEY` would have to be supplied.
+
+
### LLM
+Nesis supports any OpenAI compatible LLM service. By setting the `OPENAI_API_KEY` and `OPENAI_API_BASE` environment variables, you are able
+to configure which OpenAI API to use.
+
+When running Nesis in your private network you will need to provide your own private LLM service.
+
+???+ tip "Fully private LLM service"
+
+ Ametnes provides a fully managed private LLM service that runs enterily in your private network. Your data stays completely
+ ring-fenced in your private environment allowing you to achive full privacy.
+
## Backend API
TODO
diff --git a/docs/src/rbac.md b/docs/src/rbac.md
new file mode 100644
index 0000000..09ded6b
--- /dev/null
+++ b/docs/src/rbac.md
@@ -0,0 +1,239 @@
+# Authentication and Authorization
+
+All users must be _**authenticated**_ before they can use Nesis. As the user interacts with Nesis,
+the backend validates authentication for each request.
+
+Each action performed by the user after they have authenticated must be _**authorized**_ by the Nesis.
+Permission are issued by an Administrator. It important to note that they term administrator in this case means a user who has permission to issue any given permission.
+
+## Sequences
+### Authentication
+
+``` mermaid
+sequenceDiagram
+ autonumber
+ actor User
+ User->>Frontend: email, password
+ Frontend->>API: email, password
+ API->>API: Validate email, password
+ alt is valid
+ API->>Frontend: Auth token, 200
+ Frontend->>User: redirect to landing page
+ else not valid
+ API->>Frontend: 401
+ Frontend->>User: error message
+ end
+```
+
+
+### Authorization Sequence
+This sequence assumes a role has been created. As a use case, suppose the user is adding a datasource. The authentication
+sequence above would need to have been passed.
+
+``` mermaid
+sequenceDiagram
+ autonumber
+ actor User
+ User->>Frontend: Auth token, datasource
+ Frontend->>API: Auth token, datasource
+ API->>API: validate token
+ alt token is not valid
+ API->>Frontend: 401
+ Frontend->>User: error message
+ end
+ API->>API: check CREATE action on datasource
+ alt no action exists
+ API->>Frontend: 403
+ Frontend->>User: error message
+ else action exists:
+ API->>API: create datasource
+ API->>API: create tasks
+ API->>Frontend: 200, created datasource
+ Frontend->>User: sucess message
+ end
+```
+
+## Roles in Nesis
+
+### Overview
+A role is a named construct that combines a set of policies and attached to a user. The policy of the role
+indicates what actions that bearer (user) of that role is allowed to perform. A tabular description of the actors is below;
+
+| Actor | Description |
+|--------|---------------------------------------------------------------------------------------------------|
+| User | The system user and bearer of the role |
+| Policy | A set of rules that indicate that actions are permitted. By default all actions are not permitted |
+| Role | A named object with a policy attached. A role is then assigned to a user. |
+
+!!! note
+
+ If you are familiar with AWS's roles and policy, you'll notece that Nesis' rbac roles and policies
+ are similar to AWS' roles and policies.
+
+A user can be assigned one or multiple roles.
+
+### Policies
+A policy is a set of actions that are attached to the role. The actions are currently CREATE, READ, DELETE and UPDATE.
+For a user to perform any of these actions to any object within Nesis, they must have a permitted role attached to them.
+
+The objects in Nesis that require policies to operate include.
+
+| Object | Description |
+|------------|-------------------------------------------------------------------------------|
+| User | The system user |
+| Role | A role created on the system and containing policies. |
+| Datasource | A datasource that Nesis sources data from. |
+| Task | A scheduled job that runs in the background such as datasource ingestion jobs |
+| Prediction | Any user interaction with the rag engine is a prediction. |
+
+Some actions require more than policy rule. For example to add a datasource that has a cron schedule,
+the user role must permit CREATE:/datasource and CREATE:/tasks.
+
+This fine-grained control enables you to be flexible in your role based access control.
+
+A policy is simply a JSON document that is in the format
+
+```json title="policy.json" linenums="1"
+{
+ "items":
+ [
+ {
+ "action": "",
+ "resource": "