-
Notifications
You must be signed in to change notification settings - Fork 62
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into mb-exporter
- Loading branch information
Showing
49 changed files
with
1,137 additions
and
78 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,46 @@ | ||
| # Copyright (C) 2024 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| image: | ||
| repository: opea/chatqna-guardrails | ||
| tag: "latest" | ||
|
|
||
| # guardrails related config | ||
| guardrails-usvc: | ||
| enabled: true | ||
| SAFETY_GUARD_ENDPOINT: "http://{{ .Release.Name }}-tgi-guardrails" | ||
| SAFETY_GUARD_MODEL_ID: "meta-llama/Meta-Llama-Guard-2-8B" | ||
|
|
||
| # gaudi related config | ||
| tei: | ||
| image: | ||
| repository: ghcr.io/huggingface/tei-gaudi | ||
| tag: synapse_1.16 | ||
| resources: | ||
| limits: | ||
| habana.ai/gaudi: 1 | ||
| securityContext: | ||
| readOnlyRootFilesystem: false | ||
|
|
||
| tgi: | ||
| image: | ||
| repository: ghcr.io/huggingface/tgi-gaudi | ||
| tag: "2.0.1" | ||
| resources: | ||
| limits: | ||
| habana.ai/gaudi: 1 | ||
| MAX_INPUT_LENGTH: "1024" | ||
| MAX_TOTAL_TOKENS: "2048" | ||
| CUDA_GRAPHS: "" | ||
|
|
||
| tgi-guardrails: | ||
| LLM_MODEL_ID: "meta-llama/Meta-Llama-Guard-2-8B" | ||
| image: | ||
| repository: ghcr.io/huggingface/tgi-gaudi | ||
| tag: "2.0.1" | ||
| resources: | ||
| limits: | ||
| habana.ai/gaudi: 1 | ||
| MAX_INPUT_LENGTH: "1024" | ||
| MAX_TOTAL_TOKENS: "2048" | ||
| CUDA_GRAPHS: "" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Copyright (C) 2024 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| image: | ||
| repository: opea/chatqna-guardrails | ||
| tag: "latest" | ||
|
|
||
| # guardrails related config | ||
| guardrails-usvc: | ||
| enabled: true | ||
| SAFETY_GUARD_ENDPOINT: "http://{{ .Release.Name }}-tgi-guardrails" | ||
| SAFETY_GUARD_MODEL_ID: "meta-llama/Meta-Llama-Guard-2-8B" | ||
| tgi-guardrails: | ||
| LLM_MODEL_ID: "meta-llama/Meta-Llama-Guard-2-8B" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| # Patterns to ignore when building packages. | ||
| # This supports shell glob matching, relative path matching, and | ||
| # negation (prefixed with !). Only one pattern per line. | ||
| .DS_Store | ||
| # Common VCS dirs | ||
| .git/ | ||
| .gitignore | ||
| .bzr/ | ||
| .bzrignore | ||
| .hg/ | ||
| .hgignore | ||
| .svn/ | ||
| # Common backup files | ||
| *.swp | ||
| *.bak | ||
| *.tmp | ||
| *.orig | ||
| *~ | ||
| # Various IDEs | ||
| .project | ||
| .idea/ | ||
| *.tmproj | ||
| .vscode/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Copyright (C) 2024 Intel Corporation | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| apiVersion: v2 | ||
| name: guardrails-usvc | ||
| description: The Helm chart for deploying guardrails-usvc as microservice | ||
| type: application | ||
| version: 1.0.0 | ||
| appVersion: "v1.0" | ||
| dependencies: | ||
| - name: tgi | ||
| version: 1.0.0 | ||
| repository: file://../tgi | ||
| condition: autodependency.enabled |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| # guardrails-usvc | ||
|
|
||
| Helm chart for deploying LLM microservice. | ||
|
|
||
| guardrails-usvc depends on TGI, you should set TGI_LLM_ENDPOINT as tgi endpoint. | ||
|
|
||
| ## (Option1): Installing the chart separately: | ||
|
|
||
| First, you need to install the tgi chart, please refer to the [tgi](../tgi) chart for more information. Please use model `meta-llama/Meta-Llama-Guard-2-8B` during installation. | ||
|
|
||
| After you've deployted the tgi chart successfully, please run `kubectl get svc` to get the tgi service endpoint, i.e. `http://tgi`. | ||
|
|
||
| To install the chart, run the following: | ||
|
|
||
| ```console | ||
| cd GenAIInfra/helm-charts/common/guardrails-usvc | ||
| export HFTOKEN="insert-your-huggingface-token-here" | ||
| export SAFETY_GUARD_ENDPOINT="http://tgi" | ||
| export SAFETY_GUARD_MODEL_ID="meta-llama/Meta-Llama-Guard-2-8B" | ||
| helm dependency update | ||
| helm install guardrails-usvc . --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set SAFETY_GUARD_ENDPOINT=${SAFETY_GUARD_ENDPOINT} --set SAFETY_GUARD_MODEL_ID=${SAFETY_GUARD_MODEL_ID} --wait | ||
| ``` | ||
|
|
||
| ## (Option2): Installing the chart with dependencies automatically: | ||
|
|
||
| ```console | ||
| cd GenAIInfra/helm-charts/common/guardrails-usvc | ||
| export HFTOKEN="insert-your-huggingface-token-here" | ||
| helm dependency update | ||
| helm install guardrails-usvc . --set global.HUGGINGFACEHUB_API_TOKEN=${HFTOKEN} --set autodependency.enabled=true --wait | ||
| ``` | ||
|
|
||
| ## Verify | ||
|
|
||
| To verify the installation, run the command `kubectl get pod` to make sure all pods are running. | ||
|
|
||
| Then run the command `kubectl port-forward svc/guardrails-usvc 9090:9090` to expose the llm-uservice service for access. | ||
|
|
||
| Open another terminal and run the following command to verify the service if working: | ||
|
|
||
| ```console | ||
| curl http://localhost:9090/v1/guardrails \ | ||
| -X POST \ | ||
| -d '{"text":"How do you buy a tiger in the US?","parameters":{"max_new_tokens":32}}' \ | ||
| -H 'Content-Type: application/json' | ||
| ``` | ||
|
|
||
| ## Values | ||
|
|
||
| | Key | Type | Default | Description | | ||
| | ------------------------------- | ------ | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------ | | ||
| | global.HUGGINGFACEHUB_API_TOKEN | string | `""` | Your own Hugging Face API token | | ||
| | global.modelUseHostPath | string | `"/mnt/opea-models"` | Cached models directory, tgi will not download if the model is cached here. The host path "modelUseHostPath" will be mounted to container as /data directory | | ||
| | image.repository | string | `"opea/guardrails-usvc"` | | | ||
| | service.port | string | `"9090"` | | | ||
| | SAFETY_GUARD_ENDPOINT | string | `""` | LLM endpoint | | ||
| | SAFETY_GUARD_MODEL_ID | string | `"meta-llama/Meta-Llama-Guard-2-8B"` | Model ID for the underlying LLM service is using | |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| {{/* | ||
| Expand the name of the chart. | ||
| */}} | ||
| {{- define "guardrails-usvc.name" -}} | ||
| {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create a default fully qualified app name. | ||
| We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). | ||
| If release name contains chart name it will be used as a full name. | ||
| */}} | ||
| {{- define "guardrails-usvc.fullname" -}} | ||
| {{- if .Values.fullnameOverride }} | ||
| {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }} | ||
| {{- else }} | ||
| {{- $name := default .Chart.Name .Values.nameOverride }} | ||
| {{- if contains $name .Release.Name }} | ||
| {{- .Release.Name | trunc 63 | trimSuffix "-" }} | ||
| {{- else }} | ||
| {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
| {{- end }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create chart name and version as used by the chart label. | ||
| */}} | ||
| {{- define "guardrails-usvc.chart" -}} | ||
| {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Common labels | ||
| */}} | ||
| {{- define "guardrails-usvc.labels" -}} | ||
| helm.sh/chart: {{ include "guardrails-usvc.chart" . }} | ||
| {{ include "guardrails-usvc.selectorLabels" . }} | ||
| {{- if .Chart.AppVersion }} | ||
| app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} | ||
| {{- end }} | ||
| app.kubernetes.io/managed-by: {{ .Release.Service }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Selector labels | ||
| */}} | ||
| {{- define "guardrails-usvc.selectorLabels" -}} | ||
| app.kubernetes.io/name: {{ include "guardrails-usvc.name" . }} | ||
| app.kubernetes.io/instance: {{ .Release.Name }} | ||
| {{- end }} | ||
|
|
||
| {{/* | ||
| Create the name of the service account to use | ||
| */}} | ||
| {{- define "guardrails-usvc.serviceAccountName" -}} | ||
| {{- if .Values.serviceAccount.create }} | ||
| {{- default (include "guardrails-usvc.fullname" .) .Values.serviceAccount.name }} | ||
| {{- else }} | ||
| {{- default "default" .Values.serviceAccount.name }} | ||
| {{- end }} | ||
| {{- end }} |
Oops, something went wrong.