-
Notifications
You must be signed in to change notification settings - Fork 6
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* docs: added documentation for Google, HuggingFace and LangChain * docs: added assets for documentation * chore: removed assets to avoid CI issues --------- Signed-off-by: Kannav02 <[email protected]> Co-authored-by: Song Luar <[email protected]>
- Loading branch information
Showing
1 changed file
with
116 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,116 @@ | ||
# Backend For ORAssistant | ||
|
||
This folder contains the backend code for the ORAssistant. Follow the instructions below to set up the environment and run the backend. | ||
|
||
## Preparing the Environment Variables | ||
|
||
To configure the application, you'll need to set up the environment variables in your `.env` file: | ||
|
||
The given command would copy the template for environment variables to a local `.env` file | ||
|
||
|
||
``` | ||
cp .env.example .env | ||
``` | ||
|
||
|
||
### Setting Up Google API Key and Credentials Variables | ||
|
||
There are 2 variables that needs to be set up | ||
|
||
- `GOOGLE_API_KEY` | ||
|
||
This key is used to access the various google cloud functions. | ||
- Go to [Google Cloud Console](https://console.cloud.google.com/) | ||
- Create new project or select existing one | ||
- Enable required APIs: | ||
- Google Gemini API | ||
- Vertex AI API | ||
- Go to APIs & Services > Credentials | ||
- Click "Create Credentials" > "API Key" | ||
- Copy the generated key and it to the `.env` file | ||
|
||
- `GOOGLE_APPLICATION_CREDENTIALS` | ||
Since most of the GCP functions / services would be used by our app, we need to have a special credential that would allow `ORAssistant`'s access to the GCP | ||
Steps to set up Service Account Credentials: | ||
- In Google Cloud Console, go to IAM & Admin > Service Accounts | ||
- Click "Create Service Account" | ||
Fill in service account details | ||
- Grant required roles: | ||
- Vertex AI User | ||
- Vertex AI Service Agent | ||
- Create key (JSON format) | ||
- Download JSON file | ||
- Store securely and add path to .env: | ||
`GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json` | ||
|
||
**NOTE**: The user might need billing to be set up on google cloud account and make sure to name the file as `credentials.json` as this would be ignored by `.git` and wouldn't be exposed on Github | ||
|
||
### Setting Up LangChain Variables | ||
|
||
There are 4 variables that needs to be set up | ||
|
||
- `LANGCHAIN_TRACING_V2` | ||
|
||
This is used to enable LangChain's debugging and monitoring features, | ||
can be set to either `true` or `false` | ||
|
||
- `LANGCHAIN_ENDPOINT` | ||
|
||
The URL endpoint for LangSmith (LangChain's monitoring platform). | ||
Default value should be `https://api.smith.langchain.com` for cloud-hosted LangSmith. | ||
Used to send trace data, metrics, and debugging information from your LangChain applications. | ||
|
||
- `LANGCHAIN_API_KEY` | ||
|
||
API key required to authenticate with LangSmith platform. | ||
- Get your key from: https://smith.langchain.com/ | ||
- Create account if you don't have one | ||
- Navigate to Settings > API Keys | ||
- Create new API key | ||
- Format: starts with `lsv2_` followed by a unique string | ||
|
||
- `LANGCHAIN_PROJECT` | ||
|
||
Project identifier in LangSmith to organize and track your traces. | ||
- Create new project in LangSmith dashboard | ||
- Use the project name or ID provided | ||
- Example: "my-rag-project" | ||
- Helps organize different applications/environments | ||
- Multiple apps can share same project | ||
|
||
|
||
### Setting Up Huggingface User Access Token | ||
|
||
To set up the `HF_TOKEN` variable in `.env` file , go through the following instructions: | ||
|
||
- Go the official website for [Huggingface](https://huggingface.co/) and either Login or Sign up. | ||
- On the main page click on user access token | ||
- Click on create access token | ||
- Provide only Read Instruction for the token and Click on Generate Token | ||
|
||
|
||
|
||
|
||
Provide the value for `HF_TOKEN` with the token that is generated | ||
|
||
## Running the Application | ||
|
||
### Install Required Packages | ||
|
||
Ensure you have the necessary dependencies installed by running: | ||
|
||
```bash | ||
pip install -r requirements.txt | ||
``` | ||
### Docker Command | ||
|
||
If you want to run an isolated container for backend, you can use the following command | ||
|
||
```bash | ||
docker build -t (image_name) . | ||
``` | ||
|
||
Make sure you are in the backend folder before running the above command. | ||
|
||
**NOTE**: The project does support a `docker-compose` file that would run all of the containers together |
1441a9f
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
===================================
==> Dataset: EDA Corpus
==> Running tests for agent-retriever
/home/luarss/actions-runner/_work/ORAssistant/ORAssistant/evaluation/.venv/lib/python3.12/site-packages/deepeval/init.py:49: UserWarning: You are using deepeval version 1.4.9, however version 2.1.7 is available. You should consider upgrading via the "pip install --upgrade deepeval" command.
warnings.warn(
Fetching 2 files: 0%| | 0/2 [00:00<?, ?it/s]
Fetching 2 files: 50%|█████ | 1/2 [00:00<00:00, 7.99it/s]
Fetching 2 files: 100%|██████████| 2/2 [00:00<00:00, 9.52it/s]
Evaluating: 0%| | 0/100 [00:00<?, ?it/s]
Evaluating: 1%| | 1/100 [00:14<24:15, 14.70s/it]
Evaluating: 2%|▏ | 2/100 [00:25<20:06, 12.31s/it]
Evaluating: 3%|▎ | 3/100 [00:39<21:29, 13.29s/it]
Evaluating: 4%|▍ | 4/100 [00:49<18:58, 11.86s/it]
Evaluating: 5%|▌ | 5/100 [01:00<18:32, 11.72s/it]
Evaluating: 6%|▌ | 6/100 [01:11<17:52, 11.41s/it]
Evaluating: 7%|▋ | 7/100 [01:24<18:31, 11.96s/it]
Evaluating: 8%|▊ | 8/100 [01:42<21:03, 13.73s/it]
Evaluating: 9%|▉ | 9/100 [01:58<22:05, 14.57s/it]
Evaluating: 10%|█ | 10/100 [02:09<20:14, 13.50s/it]
Evaluating: 11%|█ | 11/100 [02:20<18:44, 12.63s/it]
Evaluating: 12%|█▏ | 12/100 [02:30<17:07, 11.68s/it]
Evaluating: 13%|█▎ | 13/100 [02:42<17:09, 11.83s/it]
Evaluating: 14%|█▍ | 14/100 [02:54<16:58, 11.84s/it]
Evaluating: 15%|█▌ | 15/100 [03:06<16:58, 11.98s/it]
Evaluating: 16%|█▌ | 16/100 [03:18<16:42, 11.93s/it]
Evaluating: 17%|█▋ | 17/100 [03:30<16:40, 12.06s/it]
Evaluating: 18%|█▊ | 18/100 [03:40<15:47, 11.55s/it]
Evaluating: 19%|█▉ | 19/100 [03:50<14:55, 11.06s/it]
Evaluating: 20%|██ | 20/100 [04:01<14:42, 11.03s/it]
Evaluating: 21%|██ | 21/100 [04:12<14:29, 11.00s/it]
Evaluating: 22%|██▏ | 22/100 [04:25<14:49, 11.41s/it]
Evaluating: 23%|██▎ | 23/100 [04:36<14:37, 11.39s/it]
Evaluating: 24%|██▍ | 24/100 [04:46<13:57, 11.02s/it]
Evaluating: 25%|██▌ | 25/100 [04:56<13:12, 10.57s/it]
Evaluating: 26%|██▌ | 26/100 [05:05<12:38, 10.25s/it]
Evaluating: 27%|██▋ | 27/100 [05:16<12:49, 10.55s/it]
Evaluating: 28%|██▊ | 28/100 [05:28<13:04, 10.90s/it]
Evaluating: 29%|██▉ | 29/100 [05:39<12:47, 10.81s/it]
Evaluating: 30%|███ | 30/100 [05:50<12:50, 11.01s/it]
Evaluating: 31%|███ | 31/100 [06:00<12:20, 10.73s/it]
Evaluating: 32%|███▏ | 32/100 [06:13<12:53, 11.38s/it]
Evaluating: 33%|███▎ | 33/100 [06:24<12:37, 11.31s/it]
Evaluating: 34%|███▍ | 34/100 [06:36<12:39, 11.50s/it]
Evaluating: 35%|███▌ | 35/100 [06:51<13:24, 12.38s/it]
Evaluating: 36%|███▌ | 36/100 [07:01<12:36, 11.81s/it]
Evaluating: 37%|███▋ | 37/100 [07:11<11:56, 11.37s/it]
Evaluating: 38%|███▊ | 38/100 [07:22<11:33, 11.19s/it]
Evaluating: 39%|███▉ | 39/100 [07:32<10:56, 10.76s/it]
Evaluating: 40%|████ | 40/100 [07:41<10:20, 10.35s/it]
Evaluating: 41%|████ | 41/100 [07:53<10:36, 10.79s/it]
Evaluating: 42%|████▏ | 42/100 [08:04<10:23, 10.75s/it]
Evaluating: 43%|████▎ | 43/100 [08:16<10:40, 11.23s/it]
Evaluating: 44%|████▍ | 44/100 [08:29<10:48, 11.59s/it]
Evaluating: 45%|████▌ | 45/100 [08:41<10:55, 11.92s/it]
Evaluating: 46%|████▌ | 46/100 [08:53<10:42, 11.89s/it]
Evaluating: 47%|████▋ | 47/100 [09:05<10:33, 11.96s/it]
Evaluating: 48%|████▊ | 48/100 [09:17<10:25, 12.03s/it]
Evaluating: 49%|████▉ | 49/100 [09:28<09:48, 11.55s/it]
Evaluating: 50%|█████ | 50/100 [09:40<09:52, 11.86s/it]
Evaluating: 51%|█████ | 51/100 [09:50<09:10, 11.24s/it]
Evaluating: 52%|█████▏ | 52/100 [10:03<09:17, 11.62s/it]
Evaluating: 53%|█████▎ | 53/100 [10:14<08:55, 11.39s/it]
Evaluating: 54%|█████▍ | 54/100 [10:26<08:51, 11.55s/it]
Evaluating: 55%|█████▌ | 55/100 [10:37<08:34, 11.43s/it]
Evaluating: 56%|█████▌ | 56/100 [10:47<08:06, 11.05s/it]
Evaluating: 57%|█████▋ | 57/100 [10:58<07:55, 11.06s/it]
Evaluating: 58%|█████▊ | 58/100 [11:09<07:40, 10.98s/it]
Evaluating: 59%|█████▉ | 59/100 [11:19<07:27, 10.91s/it]
Evaluating: 60%|██████ | 60/100 [11:32<07:37, 11.43s/it]
Evaluating: 61%|██████ | 61/100 [11:42<07:09, 11.01s/it]
Evaluating: 62%|██████▏ | 62/100 [11:53<06:56, 10.96s/it]
Evaluating: 63%|██████▎ | 63/100 [12:07<07:23, 11.99s/it]
Evaluating: 64%|██████▍ | 64/100 [12:18<06:53, 11.50s/it]
Evaluating: 65%|██████▌ | 65/100 [12:34<07:29, 12.85s/it]
Evaluating: 66%|██████▌ | 66/100 [12:46<07:11, 12.69s/it]
Evaluating: 67%|██████▋ | 67/100 [12:57<06:43, 12.22s/it]
Evaluating: 68%|██████▊ | 68/100 [13:07<06:06, 11.46s/it]
Evaluating: 69%|██████▉ | 69/100 [13:18<05:51, 11.34s/it]
Evaluating: 70%|███████ | 70/100 [13:28<05:29, 10.99s/it]
Evaluating: 71%|███████ | 71/100 [13:40<05:23, 11.14s/it]
Evaluating: 72%|███████▏ | 72/100 [13:50<05:06, 10.96s/it]
Evaluating: 73%|███████▎ | 73/100 [14:02<05:07, 11.38s/it]
Evaluating: 74%|███████▍ | 74/100 [14:13<04:51, 11.22s/it]
Evaluating: 75%|███████▌ | 75/100 [14:25<04:43, 11.34s/it]
Evaluating: 76%|███████▌ | 76/100 [14:36<04:31, 11.31s/it]
Evaluating: 77%|███████▋ | 77/100 [14:48<04:23, 11.46s/it]
Evaluating: 78%|███████▊ | 78/100 [14:59<04:08, 11.27s/it]
Evaluating: 79%|███████▉ | 79/100 [15:10<03:53, 11.13s/it]
Evaluating: 80%|████████ | 80/100 [15:21<03:41, 11.05s/it]
Evaluating: 81%|████████ | 81/100 [15:31<03:25, 10.83s/it]
Evaluating: 82%|████████▏ | 82/100 [15:41<03:12, 10.71s/it]
Evaluating: 83%|████████▎ | 83/100 [15:52<03:00, 10.60s/it]
Evaluating: 84%|████████▍ | 84/100 [16:02<02:48, 10.54s/it]
Evaluating: 85%|████████▌ | 85/100 [16:17<02:57, 11.84s/it]
Evaluating: 86%|████████▌ | 86/100 [16:29<02:46, 11.90s/it]
Evaluating: 87%|████████▋ | 87/100 [16:39<02:28, 11.42s/it]
Evaluating: 88%|████████▊ | 88/100 [16:50<02:16, 11.35s/it]
Evaluating: 89%|████████▉ | 89/100 [17:01<02:03, 11.23s/it]
Evaluating: 90%|█████████ | 90/100 [17:15<01:59, 11.90s/it]
Evaluating: 91%|█████████ | 91/100 [17:29<01:53, 12.59s/it]
Evaluating: 92%|█████████▏| 92/100 [17:40<01:36, 12.05s/it]
Evaluating: 93%|█████████▎| 93/100 [17:54<01:28, 12.63s/it]
Evaluating: 94%|█████████▍| 94/100 [18:04<01:11, 11.99s/it]
Evaluating: 95%|█████████▌| 95/100 [18:15<00:58, 11.68s/it]
Evaluating: 96%|█████████▌| 96/100 [18:26<00:45, 11.27s/it]
Evaluating: 97%|█████████▋| 97/100 [18:36<00:33, 11.10s/it]
Evaluating: 98%|█████████▊| 98/100 [18:48<00:22, 11.42s/it]
Evaluating: 99%|█████████▉| 99/100 [19:00<00:11, 11.33s/it]
Evaluating: 100%|██████████| 100/100 [19:12<00:00, 11.61s/it]
Evaluating: 100%|██████████| 100/100 [19:12<00:00, 11.52s/it]
✨ You're running DeepEval's latest Contextual Precision Metric! (using
gemini-1.5-pro-002, strict=False, async_mode=True)...
✨ You're running DeepEval's latest Contextual Recall Metric! (using
gemini-1.5-pro-002, strict=False, async_mode=True)...
✨ You're running DeepEval's latest Hallucination Metric! (using
gemini-1.5-pro-002, strict=False, async_mode=True)...
Evaluating 100 test case(s) in parallel: | | 0% (0/100) [Time Taken: 00:00, ?test case/s]
‼️ Friendly reminder 😇: You can also run evaluations with ALL of deepeval's
Evaluating 100 test case(s) in parallel: | | 1% (1/100) [Time Taken: 00:11, 11.30s/test case]
Evaluating 100 test case(s) in parallel: |▏ | 2% (2/100) [Time Taken: 00:14, 6.47s/test case]
Evaluating 100 test case(s) in parallel: |▌ | 5% (5/100) [Time Taken: 00:15, 2.03s/test case]
Evaluating 100 test case(s) in parallel: |▌ | 6% (6/100) [Time Taken: 00:15, 1.54s/test case]
Evaluating 100 test case(s) in parallel: |▉ | 9% (9/100) [Time Taken: 00:15, 1.22test case/s]
Evaluating 100 test case(s) in parallel: |█ | 10% (10/100) [Time Taken: 00:15, 1.44test case/s]
Evaluating 100 test case(s) in parallel: |█▏ | 12% (12/100) [Time Taken: 00:16, 1.83test case/s]
Evaluating 100 test case(s) in parallel: |█▌ | 15% (15/100) [Time Taken: 00:16, 2.72test case/s]
Evaluating 100 test case(s) in parallel: |█▊ | 18% (18/100) [Time Taken: 00:17, 3.04test case/s]
Evaluating 100 test case(s) in parallel: |██▎ | 23% (23/100) [Time Taken: 00:17, 5.47test case/s]
Evaluating 100 test case(s) in parallel: |██▌ | 25% (25/100) [Time Taken: 00:18, 6.23test case/s]
Evaluating 100 test case(s) in parallel: |██▊ | 28% (28/100) [Time Taken: 00:18, 8.05test case/s]
Evaluating 100 test case(s) in parallel: |███ | 30% (30/100) [Time Taken: 00:18, 6.98test case/s]
Evaluating 100 test case(s) in parallel: |███▏ | 32% (32/100) [Time Taken: 00:18, 7.67test case/s]
Evaluating 100 test case(s) in parallel: |███▍ | 34% (34/100) [Time Taken: 00:20, 3.81test case/s]
Evaluating 100 test case(s) in parallel: |███▌ | 35% (35/100) [Time Taken: 00:21, 2.51test case/s]
Evaluating 100 test case(s) in parallel: |███▌ | 36% (36/100) [Time Taken: 00:21, 2.44test case/s]
Evaluating 100 test case(s) in parallel: |███▋ | 37% (37/100) [Time Taken: 00:21, 2.62test case/s]
Evaluating 100 test case(s) in parallel: |███▊ | 38% (38/100) [Time Taken: 00:22, 2.41test case/s]
Evaluating 100 test case(s) in parallel: |███▉ | 39% (39/100) [Time Taken: 00:23, 2.04test case/s]
Evaluating 100 test case(s) in parallel: |████▏ | 42% (42/100) [Time Taken: 00:23, 3.83test case/s]
Evaluating 100 test case(s) in parallel: |████▌ | 45% (45/100) [Time Taken: 00:23, 5.89test case/s]
Evaluating 100 test case(s) in parallel: |████▊ | 48% (48/100) [Time Taken: 00:23, 8.04test case/s]
Evaluating 100 test case(s) in parallel: |█████ | 50% (50/100) [Time Taken: 00:23, 9.50test case/s]
Evaluating 100 test case(s) in parallel: |█████▏ | 52% (52/100) [Time Taken: 00:23, 10.97test case/s]
Evaluating 100 test case(s) in parallel: |█████▍ | 54% (54/100) [Time Taken: 00:24, 5.37test case/s]
Evaluating 100 test case(s) in parallel: |█████▌ | 56% (56/100) [Time Taken: 00:24, 6.68test case/s]
Evaluating 100 test case(s) in parallel: |█████▊ | 58% (58/100) [Time Taken: 00:25, 6.31test case/s]
Evaluating 100 test case(s) in parallel: |██████ | 60% (60/100) [Time Taken: 00:25, 7.67test case/s]
Evaluating 100 test case(s) in parallel: |██████▏ | 62% (62/100) [Time Taken: 00:25, 9.02test case/s]
Evaluating 100 test case(s) in parallel: |██████▍ | 64% (64/100) [Time Taken: 00:25, 8.90test case/s]
Evaluating 100 test case(s) in parallel: |██████▌ | 66% (66/100) [Time Taken: 00:25, 9.35test case/s]
Evaluating 100 test case(s) in parallel: |██████▊ | 68% (68/100) [Time Taken: 00:25, 9.67test case/s]
Evaluating 100 test case(s) in parallel: |███████ | 70% (70/100) [Time Taken: 00:27, 4.27test case/s]
Evaluating 100 test case(s) in parallel: |███████ | 71% (71/100) [Time Taken: 00:27, 3.84test case/s]
Evaluating 100 test case(s) in parallel: |███████▏ | 72% (72/100) [Time Taken: 00:27, 3.47test case/s]
Evaluating 100 test case(s) in parallel: |███████▍ | 74% (74/100) [Time Taken: 00:28, 4.40test case/s]
Evaluating 100 test case(s) in parallel: |███████▌ | 75% (75/100) [Time Taken: 00:30, 1.72test case/s]
Evaluating 100 test case(s) in parallel: |███████▌ | 76% (76/100) [Time Taken: 00:30, 1.79test case/s]
Evaluating 100 test case(s) in parallel: |███████▋ | 77% (77/100) [Time Taken: 00:30, 2.04test case/s]
Evaluating 100 test case(s) in parallel: |███████▊ | 78% (78/100) [Time Taken: 00:30, 2.49test case/s]
Evaluating 100 test case(s) in parallel: |███████▉ | 79% (79/100) [Time Taken: 00:31, 2.25test case/s]
Evaluating 100 test case(s) in parallel: |████████ | 81% (81/100) [Time Taken: 00:31, 3.46test case/s]
Evaluating 100 test case(s) in parallel: |████████▎ | 83% (83/100) [Time Taken: 00:31, 4.63test case/s]
Evaluating 100 test case(s) in parallel: |████████▍ | 84% (84/100) [Time Taken: 00:32, 4.42test case/s]
Evaluating 100 test case(s) in parallel: |████████▌ | 85% (85/100) [Time Taken: 00:32, 4.62test case/s]
Evaluating 100 test case(s) in parallel: |████████▌ | 86% (86/100) [Time Taken: 00:32, 4.10test case/s]
Evaluating 100 test case(s) in parallel: |████████▊ | 88% (88/100) [Time Taken: 00:33, 3.84test case/s]
Evaluating 100 test case(s) in parallel: |████████▉ | 89% (89/100) [Time Taken: 00:33, 4.08test case/s]
Evaluating 100 test case(s) in parallel: |█████████ | 90% (90/100) [Time Taken: 00:33, 4.64test case/s]
Evaluating 100 test case(s) in parallel: |█████████▏| 92% (92/100) [Time Taken: 00:33, 5.86test case/s]
Evaluating 100 test case(s) in parallel: |█████████▎| 93% (93/100) [Time Taken: 00:34, 3.58test case/s]
Evaluating 100 test case(s) in parallel: |█████████▍| 94% (94/100) [Time Taken: 00:35, 2.73test case/s]
Evaluating 100 test case(s) in parallel: |█████████▌| 95% (95/100) [Time Taken: 00:36, 1.79test case/s]
Evaluating 100 test case(s) in parallel: |█████████▌| 96% (96/100) [Time Taken: 00:36, 2.21test case/s]
Evaluating 100 test case(s) in parallel: |█████████▋| 97% (97/100) [Time Taken: 00:36, 1.95test case/s]
Evaluating 100 test case(s) in parallel: |█████████▊| 98% (98/100) [Time Taken: 00:37, 2.24test case/s]
Evaluating 100 test case(s) in parallel: |█████████▉| 99% (99/100) [Time Taken: 00:37, 2.30test case/s]
Evaluating 100 test case(s) in parallel: |██████████|100% (100/100) [Time Taken: 00:39, 1.10test case/s]
Evaluating 100 test case(s) in parallel: |██████████|100% (100/100) [Time Taken: 00:39, 2.52test case/s]
✓ Tests finished 🎉! Run 'deepeval login' to save and analyze evaluation results
on Confident AI.
metrics directly on Confident AI instead.
Average Metric Scores:
Contextual Precision 0.7209007936507936
Contextual Recall 0.8573571428571428
Hallucination 0.5286605339105339
Metric Passrates:
Contextual Precision 0.68
Contextual Recall 0.82
Hallucination 0.54