[k8s ChatQnA] retriever_deploy deployment failed due to "http://tei-embedding-svc.default.svc.cluster.local:6006/**" failed to access

[moting9]

Dear experts:
I followed these steps to run k8s chatQnA example.
https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/docker/xeon/README.md
https://github.com/opea-project/GenAIExamples/blob/main/ChatQnA/kubernetes/manifests/README.md

retriever-deploy deployment shows " requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/"

I have added proxy config below in "qna_configmap_xeon.yaml" and made sure tei-embedding, tei-reranking, tgi-service successfully downloaded 3 models.
http_proxy: http://proxy.xxx.xxxxx.com:xxx
https_proxy: http://proxy.xxx.xxxxx.com:xxx
no_proxy: 10.0.0.0/8,192.168.0.0/16,127.0.0.1,localhost,xxxxx.com

I suspect network or proxy config relevant issue. any suggestions are appreciated.
Below are logs.

NAMESPACE NAME READY STATUS RESTARTS AGE
default chaqna-xeon-backend-server-deploy-79cbbb7b-4q9t6 1/1 Running 0 58m
default embedding-deploy-7bb6df68f5-4k8kr 1/1 Running 0 58m
default llm-deploy-5c85678ddd-sjbsn 1/1 Running 0 58m
default redis-vector-db-6db798f98d-56vqz 1/1 Running 0 58m
default reranking-deploy-87c5bf6cd-r4cph 1/1 Running 0 58m
default retriever-deploy-6bb494f5bd-9ggnd 0/1 CrashLoopBackOff 14 ( ago) 58m
default tei-embedding-service-deploy-78fbbcbf67-xvx9v 1/1 Running 0 58m
default tei-reranking-service-deploy-6cdb544d49-fmfj8 1/1 Running 0 58m
default tgi-service-deploy-78f488ff9f-tlmg8 1/1 Running 0 58m

kubectl get svc -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
chaqna-xeon-backend-server-svc NodePort 10.103.16.9 8888:32441/TCP 69m app=chaqna-xeon-backend-server-deploy
embedding-svc ClusterIP 10.104.135.74 6000/TCP 69m app=embedding-deploy
kubernetes ClusterIP 10.96.0.1 443/TCP 6d
llm-svc ClusterIP 10.104.216.23 9000/TCP 69m app=llm-deploy
redis-vector-db ClusterIP 10.103.250.80 6379/TCP,8001/TCP 69m app=redis-vector-db
reranking-svc ClusterIP 10.103.2.220 8000/TCP 69m app=reranking-deploy
retriever-svc ClusterIP 10.110.168.101 7000/TCP 69m app=retriever-deploy
tei-embedding-svc ClusterIP 10.103.168.217 6006/TCP 69m app=tei-embedding-service-deploy
tei-reranking-svc ClusterIP 10.107.35.174 8808/TCP 69m app=tei-reranking-service-deploy
tgi-svc ClusterIP 10.104.172.25 9009/TCP 69m app=tgi-service-deploy

kubectl logs retriever-deploy-6bb494f5bd-9ggnd
Neither CUDA nor MPS are available - defaulting to CPU. Note: This module is much faster with a GPU.
Downloading detection model, please wait. This may take several minutes depending upon your network connection.
Parsing 10k filing doc for NIKE data/nke-10k-2023.pdf
Progress: |███████████████████████████████▒Downloading recognition model, please wait. This may take several minutes depending upon your network connection.
Progress: |██████████████████████████████████████████████████| 100.0% CompleteDone preprocessing. Created 270 chunks of the original pdf
Traceback (most recent call last):
File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 304, in hf_raise_for_status
response.raise_for_status()
File "/usr/local/lib/python3.11/site-packages/requests/models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/user/comps/retrievers/langchain/redis/ingest.py", line 99, in
ingest_documents()
File "/home/user/comps/retrievers/langchain/redis/ingest.py", line 88, in ingest_documents
_ = Redis.from_texts(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 485, in from_texts
instance, _ = cls.from_texts_return_keys(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 418, in from_texts_return_keys
keys = instance.add_texts(texts, metadatas, keys=keys)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 705, in add_texts
embeddings = embeddings or self._embeddings.embed_documents(list(texts))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/embeddings/huggingface_hub.py", line 95, in embed_documents
responses = self.client.post(
^^^^^^^^^^^^^^^^^
File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/inference/_client.py", line 273, in post
hf_raise_for_status(response)
File "/home/user/.local/lib/python3.11/site-packages/huggingface_hub/utils/_errors.py", line 371, in hf_raise_for_status
raise HfHubHTTPError(str(e), response=response) from e
huggingface_hub.utils._errors.HfHubHTTPError: 504 Server Error: Gateway Timeout for url: http://tei-embedding-svc.default.svc.cluster.local:6006/
/usr/local/lib/python3.11/site-packages/pydantic/_internal/fields.py:149: UserWarning: Field "model_name_or_path" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = ().
warnings.warn(
[2024-06-24 05:15:52,520] [ INFO] - CORS is enabled.
[2024-06-24 05:15:52,521] [ INFO] - Setting up HTTP server
[2024-06-24 05:15:52,521] [ INFO] - Uvicorn server setup on port 7000
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:7000 (Press CTRL+C to quit)
[2024-06-24 05:15:52,533] [ INFO] - HTTP server setup successful
Traceback (most recent call last):
File "/home/user/comps/retrievers/langchain/redis/retriever_redis.py", line 71, in
vector_db = Redis.from_existing_index(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/langchain_community/vectorstores/redis/base.py", line 562, in from_existing_index
raise ValueError(
ValueError: Redis failed to connect: Index rag-redis does not exist.

[lianhao]

Please add ".default.svc.cluster.local" to the no_proxy config too

[moting9]

Please add ".default.svc.cluster.local" to the no_proxy config too

Thanks very much! It fixed this problem.

I met with a new problem "Internal Server Error" when accessing the RAG service.:
curl http://10.97.233.140:8888/v1/chatqna -H "Content-Type: application/json" -d '{
"messages": "What is the revenue of Nike in 2023?"
}'
Internal Server Error

My GenAIExamples repo version is:
_commit a0b94b5 (HEAD -> main, origin/main, origin/HEAD)
Author: Ying Chun Guo [email protected]
Date: Wed Jun 19 15:24:27 2024 +0800

Add image build job in docker compose e2e xeon test in CI (#304)_

My GenAIComp version is
_commit e77190c (HEAD -> main, origin/main, origin/HEAD)
Author: lvliang-intel [email protected]
Date: Thu Jun 20 09:10:58 2024 +0800

Fix llm tgi test issue (#217)

* add preload env in run.sh_

kubectl logs chaqna-xeon-backend-server-deploy-79cbbb7b-vb9lx
/usr/local/lib/python3.11/site-packages/pydantic/_internal/fields.py:160: UserWarning: Field "model_name_or_path" has conflict with protected namespace "model".

You may be able to resolve this warning by setting model_config['protected_namespaces'] = ().
warnings.warn(
[2024-06-25 01:21:22,788] [ INFO] - CORS is enabled.
[2024-06-25 01:21:22,788] [ INFO] - Setting up HTTP server
[2024-06-25 01:21:22,789] [ INFO] - Uvicorn server setup on port 8888
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8888 (Press CTRL+C to quit)
[2024-06-25 01:21:22,825] [ INFO] - HTTP server setup successful
504
INFO: 10.244.0.0:33203 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request
result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule
response, node = await done_task
^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute
return await response.json(), cur_node
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json
raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings')
504
INFO: 10.244.0.0:6929 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request
result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule
response, node = await done_task
^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute
return await response.json(), cur_node
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json
raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings')
504
INFO: 10.244.0.0:52013 - "POST /v1/chatqna HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/httptools_impl.py", line 399, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 756, in call
await self.middleware_stack(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
await route.handle(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
await self.app(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
response = await func(request)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
raw_response = await run_endpoint_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/gateway.py", line 121, in handle_request
result_dict = await self.megaservice.schedule(initial_inputs={"text": prompt}, llm_parameters=parameters)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 49, in schedule
response, node = await done_task
^^^^^^^^^^^^^^^
File "/home/user/GenAIComps/comps/cores/mega/orchestrator.py", line 98, in execute
return await response.json(), cur_node
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1176, in json
raise ContentTypeError(
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('http://embedding-svc:6000/v1/embeddings')

[lianhao]

please add those XXX_SERVICE_HOST_IP's values into no_proxy too, e.g. embedding-svc,retriever-svc, etc.

[moting9]

Thanks! the solution works.