Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support multiple nodes of a graph, apply different endpoint env for different node #176

Closed
wants to merge 37 commits into from
Closed

support multiple nodes of a graph, apply different endpoint env for different node #176

Show file tree
Hide file tree
Changes from 27 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
18ca0a2
support multiple nodes in graph
KfreeZ Jul 16, 2024
b4590ef
clean up code
KfreeZ Jul 17, 2024
f220c2e
seperate reconcile logic of resource service and router service
KfreeZ Jul 17, 2024
3037304
Merge remote-tracking branch 'upstream/main' into multiEnvVar
KfreeZ Jul 17, 2024
0b4dbe0
adopt new manifests
KfreeZ Jul 17, 2024
540a4a2
modify e2e scripts and ut tests to fit in new manifests
KfreeZ Jul 17, 2024
46bd4d6
modify ut tests to fit in new manifests
KfreeZ Jul 17, 2024
901cb09
support multiple nodes in graph
KfreeZ Jul 16, 2024
05f82d3
clean up code
KfreeZ Jul 17, 2024
b52f044
seperate reconcile logic of resource service and router service
KfreeZ Jul 17, 2024
afa16d5
Add tts and speecht5 components helm chart
yongfengdu Jul 16, 2024
fc48011
Update the script to generate comp manifest
lianhao Jul 16, 2024
8388e08
helm: remove unused Probes
lianhao Jul 16, 2024
135c59b
helm: Add tei-gaudi support
lianhao Jul 17, 2024
aa17ba3
helm redis-vector-db: Add missings in value file
lianhao Jul 17, 2024
4d68983
helm: Use empty string instead of null in value files
lianhao Jul 17, 2024
7dad3c4
Add component k8s manifest files
lianhao Jul 17, 2024
f01cf4a
Add helm test for chart redis-vector-db
lianhao Jul 16, 2024
ea10be2
Add helm test for chart tgi
lianhao Jul 16, 2024
334578d
Add helm test for chart tei
lianhao Jul 16, 2024
b5a40c1
Add helm test for chart teirerank
lianhao Jul 16, 2024
d7b93c5
helm test: Make curl fail if http_status > 400 returned
lianhao Jul 16, 2024
16c7965
adopt new manifests
KfreeZ Jul 17, 2024
83e8e11
modify e2e scripts and ut tests to fit in new manifests
KfreeZ Jul 17, 2024
1204dea
modify ut tests to fit in new manifests
KfreeZ Jul 17, 2024
201d942
Merge branch 'multiEnvVar' of https://github.com/KfreeZ/GenAIInfra in…
KfreeZ Jul 17, 2024
d894673
Merge branch 'main' into multiEnvVar
KfreeZ Jul 17, 2024
69d2391
add TEI endpoint for retriver
KfreeZ Jul 18, 2024
a8b53d0
save serviceURL
KfreeZ Jul 18, 2024
25710f4
Merge remote-tracking branch 'upstream/main' into multiEnvVar
KfreeZ Jul 19, 2024
aca4ede
use service name to identify the downstream service
KfreeZ Jul 19, 2024
f0a815e
change endpoint to service name
KfreeZ Jul 19, 2024
5acd489
fix lint error
KfreeZ Jul 19, 2024
55c25cd
fix bug
KfreeZ Jul 19, 2024
eddd911
change ENV name in GMC examples
KfreeZ Jul 19, 2024
a632837
add Web retriever
KfreeZ Jul 19, 2024
7673c71
Merge branch 'main' into multiEnvVar
KfreeZ Jul 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .github/workflows/scripts/e2e/gmc_install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ function copy_manifests() {
# Copy manifest into gmc
mkdir -p $(pwd)/config/manifests
cp $(dirname $(pwd))/manifests/ChatQnA/*.yaml -p $(pwd)/config/manifests/
cp $(dirname $(pwd))/manifests/common/*.yaml -p $(pwd)/config/manifests/
cp $(dirname $(pwd))/manifests/DocSum/xeon/docsum_llm.yaml -p $(pwd)/config/manifests/
cp $(dirname $(pwd))/manifests/DocSum/gaudi/docsum_gaudi_llm.yaml -p $(pwd)/config/manifests/
}
Expand All @@ -59,7 +60,8 @@ function init_gmc() {
sed -i "s|name: system|name: $SYSTEM_NAMESPACE|g" $(pwd)/config/rbac/gmc-manager-rbac.yaml

# replace the mount dir "path: /mnt/model" with "path: $CHART_MOUNT"
find . -name '*.yaml' -type f -exec sed -i "s#path: /mnt/models#path: $MOUNT_DIR#g" {} \;
# find . -name '*.yaml' -type f -exec sed -i "s#path: /mnt/models#path: $MOUNT_DIR#g" {} \;
find . -name '*.yaml' -type f -exec sed -i "s#path: /mnt/opea-models#path: $MOUNT_DIR#g" {} \;
# replace the repository "image: opea/*" with "image: ${IMAGE_REPO}opea/"
find . -name '*.yaml' -type f -exec sed -i "s#image: opea/*#image: ${IMAGE_REPO}opea/#g" {} \;
find . -name '*.yaml' -type f -exec sed -i "s#image: \"opea/*#image: \"${IMAGE_REPO}opea/#g" {} \;
Expand Down
4 changes: 4 additions & 0 deletions microservices-connector/config/samples/chatQnA_gaudi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ spec:
serviceName: embedding-svc
config:
endpoint: /v1/embeddings
TEI_EMBEDDING_ENDPOINT: TeiEmbedding
- name: TeiEmbeddingGaudi
internalService:
serviceName: tei-embedding-gaudi-svc
Expand All @@ -33,6 +34,7 @@ spec:
serviceName: retriever-svc
config:
endpoint: /v1/retrieval
REDIS_URL: VectorDB
- name: VectorDB
internalService:
serviceName: redis-vector-db
Expand All @@ -43,6 +45,7 @@ spec:
serviceName: reranking-svc
config:
endpoint: /v1/reranking
TEI_RERANKING_ENDPOINT: TeiReranking
- name: TeiReranking
internalService:
serviceName: tei-reranking-svc
Expand All @@ -55,6 +58,7 @@ spec:
serviceName: llm-svc
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: Tgi
- name: TgiGaudi
internalService:
serviceName: tgi-gaudi-svc
Expand Down
4 changes: 4 additions & 0 deletions microservices-connector/config/samples/chatQnA_xeon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ spec:
serviceName: embedding-svc
config:
endpoint: /v1/embeddings
TEI_EMBEDDING_ENDPOINT: TeiEmbedding
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should use servcieName, in this case tei-embedding-svc.

- name: TeiEmbedding
internalService:
serviceName: tei-embedding-svc
Expand All @@ -33,6 +34,7 @@ spec:
serviceName: retriever-svc
config:
endpoint: /v1/retrieval
REDIS_URL: VectorDB
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same here, use redis-vector-db, in this way we can support multiple endpoint for switch scenario

- name: VectorDB
internalService:
serviceName: redis-vector-db
Expand All @@ -43,6 +45,7 @@ spec:
serviceName: reranking-svc
config:
endpoint: /v1/reranking
TEI_RERANKING_ENDPOINT: TeiReranking
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same

- name: TeiReranking
internalService:
serviceName: tei-reranking-svc
Expand All @@ -55,6 +58,7 @@ spec:
serviceName: llm-svc
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: Tgi
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the same

- name: Tgi
internalService:
serviceName: tgi-service-m
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/codegen_gaudi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
serviceName: llm-service
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: TgiGaudi
- name: TgiGaudi
internalService:
serviceName: tgi-gaudi-svc
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/codegen_xeon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
serviceName: llm-service
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: Tgi
- name: Tgi
internalService:
serviceName: tgi-service
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/codetrans_gaudi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
serviceName: codetrans-service
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: TgiGaudi
- name: TgiGaudi
internalService:
serviceName: tgi-gaudi-svc
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/codetrans_xeon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ spec:
serviceName: codetrans-service
config:
endpoint: /v1/chat/completions
TGI_LLM_ENDPOINT: Tgi
- name: Tgi
internalService:
serviceName: tgi-service
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/docsum_gaudi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ spec:
HUGGING_FACE_HUB_TOKEN: <HUGGING_FACE_HUB_TOKEN>
HF_TOKEN: <HF_TOKEN>
PORT: "9009"
TGI_LLM_ENDPOINT: TgiGaudi
- name: TgiGaudi
internalService:
serviceName: tgi-gaudi-svc
Expand Down
1 change: 1 addition & 0 deletions microservices-connector/config/samples/docsum_xeon.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ spec:
HUGGING_FACE_HUB_TOKEN: <HUGGING_FACE_HUB_TOKEN>
HF_TOKEN: <HF_TOKEN>
PORT: "9009"
TGI_LLM_ENDPOINT: Tgi
- name: Tgi
internalService:
serviceName: tgi-svc
Expand Down
Loading
Loading