Add knowledge graph components (opea-project#171)

* enable ragas (opea-project#129)

Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Fix RAG performance issues (opea-project#132)

* Fix RAG performance issues

Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* add microservice level perf statistics (opea-project#135)

* add statistics

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Add More Contents to the Table of MicroService (opea-project#141)

* Add More Contents to the Table MicroService

Signed-off-by: zehao-intel <[email protected]>

* reorder

Signed-off-by: zehao-intel <[email protected]>

* Update README.md

* refine structure

Signed-off-by: zehao-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix model

Signed-off-by: zehao-intel <[email protected]>

* refine table

Signed-off-by: zehao-intel <[email protected]>

* put llm to the ground

Signed-off-by: zehao-intel <[email protected]>

---------

Signed-off-by: zehao-intel <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Use common security content for OPEA projects (opea-project#151)

* add python coverage

Signed-off-by: chensuyue <[email protected]>

* docs update

Signed-off-by: chensuyue <[email protected]>

* Revert "add python coverage"

This reverts commit 69615b1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: chensuyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Enable vLLM Gaudi support for LLM service based on officially habana vllm release (opea-project#137)

Signed-off-by: tianyil1 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* add knowledge graph

Signed-off-by: Xinyao Wang <[email protected]>

* knowledge graph microservice update

Signed-off-by: Xinyao Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Support Dataprep Microservice with Llama Index (opea-project#154)

* move file to langchain folder

Signed-off-by: letonghan <[email protected]>

* support dataprep with llama_index

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add e2e test script

Signed-off-by: letonghan <[email protected]>

* update test script name

Signed-off-by: letonghan <[email protected]>

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Support Embedding Microservice with Llama Index (opea-project#150)

* fix stream=false doesn't work issue

Signed-off-by: letonghan <[email protected]>

* support embedding comp with llama_index

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add More Contents to the Table of MicroService (opea-project#141)

* Add More Contents to the Table MicroService

Signed-off-by: zehao-intel <[email protected]>

* reorder

Signed-off-by: zehao-intel <[email protected]>

* Update README.md

* refine structure

Signed-off-by: zehao-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix model

Signed-off-by: zehao-intel <[email protected]>

* refine table

Signed-off-by: zehao-intel <[email protected]>

* put llm to the ground

Signed-off-by: zehao-intel <[email protected]>

---------

Signed-off-by: zehao-intel <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use common security content for OPEA projects (opea-project#151)

* add python coverage

Signed-off-by: chensuyue <[email protected]>

* docs update

Signed-off-by: chensuyue <[email protected]>

* Revert "add python coverage"

This reverts commit 69615b1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: chensuyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Enable vLLM Gaudi support for LLM service based on officially habana vllm release (opea-project#137)

Signed-off-by: tianyil1 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* support embedding comp with llama_index

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add test script for embedding llama_inde

Signed-off-by: letonghan <[email protected]>

* remove conflict requirements

Signed-off-by: letonghan <[email protected]>

* update test script

Signed-off-by: letonghan <[email protected]>

* udpate

Signed-off-by: letonghan <[email protected]>

* update

Signed-off-by: letonghan <[email protected]>

* update

Signed-off-by: letonghan <[email protected]>

* fix ut issue

Signed-off-by: letonghan <[email protected]>

---------

Signed-off-by: letonghan <[email protected]>
Signed-off-by: zehao-intel <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: tianyil1 <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: zehao-intel <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: Tianyi Liu <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Support Ollama microservice (opea-project#142)

* Add Ollama Support

Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Fix dataprep microservice path issue (opea-project#163)

Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* update CI to support dataprep_redis path level change (opea-project#155)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: letonghan <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Add Gateway for Translation (opea-project#169)

* add translation gateway

Signed-off-by: zehao-intel <[email protected]>

* fix import

Signed-off-by: zehao-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: zehao-intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Update LLM readme (opea-project#172)

* Update LLM readme

Signed-off-by: lvliang-intel <[email protected]>

* update readme

Signed-off-by: lvliang-intel <[email protected]>

* update tgi readme

Signed-off-by: lvliang-intel <[email protected]>

* rollback requirements.txt

Signed-off-by: lvliang-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: lvliang-intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* add milvus microservice (opea-project#158)

* Use common security content for OPEA projects (opea-project#151)

* add python coverage

Signed-off-by: chensuyue <[email protected]>

* docs update

Signed-off-by: chensuyue <[email protected]>

* Revert "add python coverage"

This reverts commit 69615b1.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: chensuyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: jinjunzh <[email protected]>

* add milvus microservice

Signed-off-by: jinjunzh <[email protected]>

* fix the typo

Signed-off-by: jinjunzh <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: jinjunzh <[email protected]>

---------

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: jinjunzh <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* enable python coverage (opea-project#149)

Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Add Ray version for multi file process (opea-project#119)

* add ray version document to redis

Signed-off-by: Chendi Xue <[email protected]>

* update test

Signed-off-by: Chendi Xue <[email protected]>

* Add test

Signed-off-by: Chendi Xue <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add TIMEOUT in container environment and return status

Signed-off-by: Chendi Xue <[email protected]>

* rebase on new folder layout

Signed-off-by: Chendi Xue <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Chendi Xue <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Add codecov (opea-project#178)

Signed-off-by: Sun, Xuehao <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Rename lm-eval folder to utils/lm-eval (opea-project#179)

Signed-off-by: changwangss <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Support rerank and retrieval of RAG OPT (opea-project#164)

* supported bce model for rerank.

Signed-off-by: Xinyu Ye <[email protected]>

* change folder

Signed-off-by: Xinyu Ye <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* change path in test file.

Signed-off-by: Xinyu Ye <[email protected]>

---------

Signed-off-by: Xinyu Ye <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Support vLLM XFT LLM microservice (opea-project#174)

* Support vLLM XFT serving

Signed-off-by: lvliang-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix access vllm issue

Signed-off-by: lvliang-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* add permission for run.sh

Signed-off-by: lvliang-intel <[email protected]>

* add readme

Signed-off-by: lvliang-intel <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fix proxy issue

Signed-off-by: lvliang-intel <[email protected]>

---------

Signed-off-by: lvliang-intel <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* Update Dataprep Microservice README (opea-project#173)

* update dataprep readme

Signed-off-by: letonghan <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: letonghan <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* fixed milvus port conflict issues during deployment (opea-project#183)

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* fixed milvus port conflict issues during deployment

* align port for unified retrieval microservice

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* remove

Signed-off-by: Xinyao Wang <[email protected]>

* remove hard address

Signed-off-by: Xinyao Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* update readme

Signed-off-by: Xinyao Wang <[email protected]>

* add example data and ingestion

Signed-off-by: Xinyao Wang <[email protected]>

* fix typ

Signed-off-by: Xinyao Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix dataprep timeout issue (opea-project#203)

Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* Add a new embedding MosecEmbedding (opea-project#182)

* Add a new embedding MosecEmbedding.

Signed-off-by: Jincheng Miao <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: Jincheng Miao <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Signed-off-by: Xinyao Wang <[email protected]>

* expand timeout for microservice test (opea-project#208)

Signed-off-by: chensuyue <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>

* fix typo

Signed-off-by: Xinyao Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Signed-off-by: Xinyao Wang <[email protected]>

* fix requirement

Signed-off-by: Xinyao Wang <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

---------

Signed-off-by: XuhuiRen <[email protected]>
Signed-off-by: Xinyao Wang <[email protected]>
Signed-off-by: lvliang-intel <[email protected]>
Signed-off-by: zehao-intel <[email protected]>
Signed-off-by: chensuyue <[email protected]>
Signed-off-by: tianyil1 <[email protected]>
Signed-off-by: letonghan <[email protected]>
Signed-off-by: jinjunzh <[email protected]>
Signed-off-by: Sun, Xuehao <[email protected]>
Signed-off-by: Chendi Xue <[email protected]>
Signed-off-by: changwangss <[email protected]>
Signed-off-by: Xinyu Ye <[email protected]>
Signed-off-by: Jincheng Miao <[email protected]>
Co-authored-by: XuhuiRen <[email protected]>
Co-authored-by: lvliang-intel <[email protected]>
Co-authored-by: Sihan Chen <[email protected]>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: zehao-intel <[email protected]>
Co-authored-by: chen, suyue <[email protected]>
Co-authored-by: Tianyi Liu <[email protected]>
Co-authored-by: Letong Han <[email protected]>
Co-authored-by: jasperzhu <[email protected]>
Co-authored-by: Chendi.Xue <[email protected]>
Co-authored-by: Sun, Xuehao <[email protected]>
Co-authored-by: Wang, Chang <[email protected]>
Co-authored-by: XinyuYe-Intel <[email protected]>
Co-authored-by: Jincheng Miao <[email protected]>
Signed-off-by: Yogesh Pandey <[email protected]>

comps/__init__.py

@@ -16,6 +16,7 @@
     TextDoc,
     RAGASParams,
     RAGASScores,
+    GraphDoc,
     LVMDoc,
 )
 

comps/cores/mega/constants.py

@@ -27,6 +27,7 @@ class ServiceType(Enum):
     UNDEFINED = 10
     RAGAS = 11
     LVM = 12
+    KNOWLEDGE_GRAPH = 13
 
 
 class MegaServiceEndpoint(Enum):
@@ -50,6 +51,8 @@ class MegaServiceEndpoint(Enum):
     RERANKING = "/v1/reranking"
     GUARDRAILS = "/v1/guardrails"
     RAGAS = "/v1/ragas"
+    GRAPHS = "/v1/graphs"
+
     # COMMON
     LIST_SERVICE = "/v1/list_service"
     LIST_PARAMETERS = "/v1/list_parameters"

comps/cores/proto/docarray.py

@@ -104,6 +104,19 @@ class RAGASScores(BaseDoc):
     context_precision: float
 
 
+class GraphDoc(BaseDoc):
+    text: str
+    strtype: Optional[str] = Field(
+        description="type of input query, can be 'query', 'cypher', 'rag'",
+        default="query",
+    )
+    max_new_tokens: Optional[int] = Field(default=1024)
+    rag_index_name: Optional[str] = Field(default="rag")
+    rag_node_label: Optional[str] = Field(default="Task")
+    rag_text_node_properties: Optional[list] = Field(default=["name", "description", "status"])
+    rag_embedding_node_property: Optional[str] = Field(default="embedding")
+
+
 class LVMDoc(BaseDoc):
     image: str
     prompt: str

comps/knowledgegraphs/README.md

@@ -0,0 +1,146 @@
+# Knowledge Graph Microservice
+
+This microservice, designed for efficiently handling and retrieving informantion from knowledge graph. The microservice integrates text retriever, knowledge graph quick search and LLM agent, which can be combined to enhance question answering.
+
+The service contains three modes:
+
+- "cypher": Query knowledge graph directly with cypher
+- "rag": Apply similarity search on embeddings of knowledge graph
+- "query": An LLM agent will automatically choose tools (RAG or CypherChain) to enhance the question answering
+
+Here is the overall workflow:
+
+![Workflow](doc/workflow.png)
+
+A prerequisite for using this microservice is that users must have a knowledge gragh database already running, and currently we have support [Neo4J](https://neo4j.com/) for quick deployment. Users need to set the graph service's endpoint into an environment variable and microservie utilizes it for data injestion and retrieve. If user want to use "rag" and "query" mode, still need a LLM text generation service (etc., TGI, vLLM and Ray) already running.
+
+Overall, this microservice provides efficient support for applications related with graph dataset, especially for answering multi-part questions, or any other conditions including comples relationship between entities.
+
+# 🚀1. Start Microservice with Docker
+
+## 1.1 Setup Environment Variables
+
+```bash
+export NEO4J_ENDPOINT="neo4j://${your_ip}:7687"
+export NEO4J_USERNAME="neo4j"
+export NEO4J_PASSWORD=${define_a_password}
+export HUGGINGFACEHUB_API_TOKEN=${your_huggingface_api_token}
+export LLM_ENDPOINT="http://${your_ip}:8080"
+export LLM_MODEL="meta-llama/Llama-2-7b-hf"
+export AGENT_LLM="HuggingFaceH4/zephyr-7b-beta"
+```
+
+## 1.2 Start Neo4j Service
+
+```bash
+docker pull neo4j
+
+docker run --rm \
+    --publish=7474:7474 --publish=7687:7687 \
+    --env NEO4J_AUTH=$NEO4J_USER/$NEO4J_PASSWORD \
+    --volume=$PWD/neo4j_data:"/data" \
+    --env='NEO4JLABS_PLUGINS=["apoc"]' \
+    neo4j
+```
+
+## 1.3 Start LLM Service for "rag"/"query" mode
+
+You can start any LLM microserve, here we take TGI as an example.
+
+```bash
+docker run -p 8080:80 \
+    -v $PWD/llm_data:/data --runtime=habana \
+    -e HABANA_VISIBLE_DEVICES=all \
+    -e OMPI_MCA_btl_vader_single_copy_mechanism=none \
+    -e HUGGING_FACE_HUB_TOKEN=$HUGGINGFACEHUB_API_TOKEN \
+    --cap-add=sys_nice \
+    --ipc=host \
+    ghcr.io/huggingface/tgi-gaudi:2.0.0 \
+    --model-id $LLM_MODEL \
+    --max-input-tokens 1024 \
+    --max-total-tokens 2048
+```
+
+Verify LLM service.
+
+```bash
+curl $LLM_ENDPOINT/generate \
+  -X POST \
+  -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":32}}' \
+  -H 'Content-Type: application/json'
+```
+
+## 1.4 Start Microservice
+
+```bash
+cd ../..
+docker build -t opea/knowledge_graphs:latest \
+    --build-arg https_proxy=$https_proxy \
+    --build-arg http_proxy=$http_proxy \
+    -f comps/knowledgegraphs/langchain/docker/Dockerfile .
+
+docker run --rm \
+    --name="knowledge-graph-server" \
+    -p 8060:8060 \
+    --ipc=host \
+    -e http_proxy=$http_proxy \
+    -e https_proxy=$https_proxy \
+    -e NEO4J_ENDPOINT=$NEO4J_ENDPOINT \
+    -e NEO4J_USERNAME=$NEO4J_USERNAME \
+    -e NEO4J_PASSWORD=$NEO4J_PASSWORD \
+    -e HUGGINGFACEHUB_API_TOKEN=$HUGGINGFACEHUB_API_TOKEN \
+    -e LLM_ENDPOINT=$LLM_ENDPOINT \
+    opea/knowledge_graphs:latest
+```
+
+# 🚀2. Consume Knowledge Graph Service
+
+## 2.1 Cypher mode
+
+```bash
+curl http://${your_ip}:8060/v1/graphs \
+  -X POST \
+  -d "{\"text\":\"MATCH (t:Task {status:'open'}) RETURN count(*)\",\"strtype\":\"cypher\"}" \
+  -H 'Content-Type: application/json'
+```
+
+Example output:
+![Cypher Output](doc/output_cypher.png)
+
+## 2.2 Rag mode
+
+```bash
+curl http://${your_ip}:8060/v1/graphs \
+  -X POST \
+  -d "{\"text\":\"How many open tickets there are?\",\"strtype\":\"rag\", \"max_new_tokens\":128}" \
+  -H 'Content-Type: application/json'
+```
+
+Example output:
+![Cypher Output](doc/output_rag.png)
+
+## 2.3 Query mode
+
+First example:
+
+```bash
+curl http://${your_ip}:8060/v1/graphs \
+  -X POST \
+  -d "{\"text\":\"Which tasks have optimization in their description?\",\"strtype\":\"query\"}" \
+  -H 'Content-Type: application/json'
+```
+
+Example output:
+![Cypher Output](doc/output_query1.png)
+
+Second example:
+
+```bash
+curl http://${your_ip}:8060/v1/graphs \
+  -X POST \
+  -d "{\"text\":\"Which team is assigned to maintain PaymentService?\",\"strtype\":\"query\"}" \
+  -H 'Content-Type: application/json'
+```
+
+Example output:
+![Cypher Output](doc/output_query2.png)

comps/knowledgegraphs/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

comps/knowledgegraphs/build_docker.sh

@@ -0,0 +1,7 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+docker build -t opea/knowledge_graphs:latest \
+    --build-arg https_proxy=$https_proxy \
+    --build-arg http_proxy=$http_proxy \
+    -f comps/knowledgegraphs/langchain/docker/Dockerfile .

comps/knowledgegraphs/langchain/__init__.py

@@ -0,0 +1,2 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0

comps/knowledgegraphs/langchain/data/microservices.json

@@ -0,0 +1,3 @@
+{
+  "query": "MERGE (catalog:Microservice {name: 'CatalogService', technology: 'Java'}) MERGE (order:Microservice {name: 'OrderService', technology: 'Python'}) MERGE (user:Microservice {name: 'UserService', technology: 'Go'}) MERGE (payment:Microservice {name: 'PaymentService', technology: 'Node.js'}) MERGE (inventory:Microservice {name: 'InventoryService', technology: 'Java'}) MERGE (shipping:Microservice {name: 'ShippingService', technology: 'Python'}) MERGE (review:Microservice {name: 'ReviewService', technology: 'Go'}) MERGE (recommendation:Microservice {name: 'RecommendationService', technology: 'Node.js'}) MERGE (auth:Microservice {name: 'AuthService', technology: 'Node.js'}) MERGE (db:Microservice {name: 'Database', technology: 'SQL'}) MERGE (cache:Microservice {name: 'Cache', technology: 'In-memory'}) MERGE (mq:Microservice {name: 'MessageQueue', technology: 'Pub-Sub'}) MERGE (api:Microservice {name: 'ExternalAPI', technology: 'REST'}) MERGE (bugFixCatalog:Task {name: 'BugFix', description: 'Address and resolve a critical bug impacting the CatalogService, affecting the user interface and experience, and hampering the overall performance and responsiveness of the service.', status: 'open'}) MERGE (featureAddOrder:Task {name: 'FeatureAdd', description: 'Implement a new feature in OrderService to facilitate bulk orders, ensuring the features seamless integration with existing functionalities and maintaining the overall stability and performance of the service.', status: 'in progress'}) MERGE (refactorUser:Task {name: 'Refactor', description: 'Refactor the UserService codebase to enhance its readability, maintainability, and scalability, focusing primarily on modularization and optimization of existing functionalities.', status: 'completed'}) MERGE (optimizePayment:Task {name: 'Optimize', description: 'Optimize PaymentService by refining the transaction processing logic, reducing the service’s latency, and improving its reliability and efficiency in handling transactions.', status: 'open'}) MERGE (updateInventory:Task {name: 'Update', description: 'Update InventoryService to include real-time stock updates, ensuring accurate reflection of the inventory levels and aiding in the efficient management of stock.', status: 'in progress'}) MERGE (enhanceShipping:Task {name: 'Enhance', description: 'Enhance the ShippingService by integrating a new shipping partner API, thereby expanding the shipping options available to the customers and improving the overall delivery experience.', status: 'completed'}) MERGE (reviewFix:Task {name: 'ReviewFix', description: 'Rectify a recurring issue in ReviewService affecting the retrieval of user reviews, by refining the service’s logic and improving its efficiency in handling and displaying user reviews.', status: 'open'}) MERGE (recommendationFeature:Task {name: 'RecommendationFeature', description: 'Add a new feature to RecommendationService to provide more personalized and accurate product recommendations to the users, leveraging user behavior and preference data.', status: 'in progress'}) MERGE (optimizeAuth:Task {name: 'Optimize', description: 'Enhance AuthService’s performance and security by optimizing the authentication mechanisms and implementing additional security measures to safeguard user information.', status: 'open'}) MERGE (newTask:Task {name: 'ImproveSecurity', description: 'Enhance the security of microservices by implementing advanced encryption and securing endpoints.', status: 'open'}) MERGE (teamA:Team {name: 'TeamA'}) MERGE (teamB:Team {name: 'TeamB'}) MERGE (teamC:Team {name: 'TeamC'}) MERGE (teamD:Team {name: 'TeamD'}) MERGE (alice:Person {name: 'Alice'}) MERGE (bob:Person {name: 'Bob'}) MERGE (charlie:Person {name: 'Charlie'}) MERGE (diana:Person {name: 'Diana'}) MERGE (eva:Person {name: 'Eva'}) MERGE (frank:Person {name: 'Frank'}) MERGE (catalog)-[:DEPENDS_ON]->(db) MERGE (order)-[:DEPENDS_ON]->(db) MERGE (user)-[:DEPENDS_ON]->(db) MERGE (payment)-[:DEPENDS_ON]->(db) MERGE (inventory)-[:DEPENDS_ON]->(db) MERGE (shipping)-[:DEPENDS_ON]->(mq) MERGE (review)-[:DEPENDS_ON]->(cache) MERGE (recommendation)-[:DEPENDS_ON]->(api) MERGE (auth)-[:DEPENDS_ON]->(db) MERGE (order)-[:DEPENDS_ON]->(inventory) MERGE (order)-[:DEPENDS_ON]->(shipping) MERGE (order)-[:DEPENDS_ON]->(payment) MERGE (catalog)-[:DEPENDS_ON]->(review) MERGE (catalog)-[:DEPENDS_ON]->(recommendation) MERGE (user)-[:DEPENDS_ON]->(auth) MERGE (payment)-[:DEPENDS_ON]->(auth) MERGE (shipping)-[:DEPENDS_ON]->(auth) MERGE (catalog)-[:MAINTAINED_BY]->(teamA) MERGE (order)-[:MAINTAINED_BY]->(teamB) MERGE (user)-[:MAINTAINED_BY]->(teamC) MERGE (payment)-[:MAINTAINED_BY]->(teamD) MERGE (inventory)-[:MAINTAINED_BY]->(teamA) MERGE (shipping)-[:MAINTAINED_BY]->(teamB) MERGE (review)-[:MAINTAINED_BY]->(teamC) MERGE (recommendation)-[:MAINTAINED_BY]->(teamD) MERGE (auth)-[:MAINTAINED_BY]->(teamA) MERGE (bugFixCatalog)-[:ASSIGNED_TO]->(teamA) MERGE (featureAddOrder)-[:ASSIGNED_TO]->(teamB) MERGE (refactorUser)-[:ASSIGNED_TO]->(teamC) MERGE (optimizePayment)-[:ASSIGNED_TO]->(teamD) MERGE (updateInventory)-[:ASSIGNED_TO]->(teamA) MERGE (enhanceShipping)-[:ASSIGNED_TO]->(teamB) MERGE (reviewFix)-[:ASSIGNED_TO]->(teamC) MERGE (recommendationFeature)-[:ASSIGNED_TO]->(teamD) MERGE (optimizeAuth)-[:ASSIGNED_TO]->(teamA) MERGE (bugFixCatalog)-[:LINKED_TO]->(catalog) MERGE (featureAddOrder)-[:LINKED_TO]->(order) MERGE (refactorUser)-[:LINKED_TO]->(user) MERGE (optimizePayment)-[:LINKED_TO]->(payment) MERGE (updateInventory)-[:LINKED_TO]->(inventory) MERGE (enhanceShipping)-[:LINKED_TO]->(shipping) MERGE (reviewFix)-[:LINKED_TO]->(review) MERGE (recommendationFeature)-[:LINKED_TO]->(recommendation) MERGE (optimizeAuth)-[:LINKED_TO]->(auth) MERGE (alice)-[:PART_OF]->(teamA) MERGE (bob)-[:PART_OF]->(teamB) MERGE (charlie)-[:PART_OF]->(teamC) MERGE (diana)-[:PART_OF]->(teamD) MERGE (eva)-[:PART_OF]->(teamA) MERGE (frank)-[:PART_OF]->(teamB) MERGE (newTask)-[:LINKED_TO]->(auth) MERGE (newTask)-[:ASSIGNED_TO]->(teamA)"
+}

comps/knowledgegraphs/langchain/docker/Dockerfile

@@ -0,0 +1,27 @@
+
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+FROM langchain/langchain:latest
+
+RUN apt-get update -y && apt-get install -y --no-install-recommends --fix-missing \
+    libgl1-mesa-glx \
+    libjemalloc-dev \
+    vim
+
+RUN useradd -m -s /bin/bash user && \
+    mkdir -p /home/user && \
+    chown -R user /home/user/
+
+COPY comps /home/user/comps
+
+USER user
+
+RUN pip install --no-cache-dir --upgrade pip && \
+    pip install --no-cache-dir -r /home/user/comps/knowledgegraphs/requirements.txt
+
+ENV PYTHONPATH=$PYTHONPATH:/home/user
+
+WORKDIR /home/user/comps/knowledgegraphs/langchain
+
+ENTRYPOINT ["python", "knowledge_graph.py"]

comps/knowledgegraphs/langchain/ingest.py

@@ -0,0 +1,21 @@
+# Copyright (C) 2024 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+
+import json
+import os
+
+from langchain_community.graphs import Neo4jGraph
+
+neo4j_endpoint = os.getenv("NEO4J_ENDPOINT", "neo4j://localhost:7687")
+neo4j_username = os.getenv("NEO4J_USERNAME", "neo4j")
+neo4j_password = os.getenv("NEO4J_PASSWORD", "neo4j")
+graph = Neo4jGraph(url=neo4j_endpoint, username=neo4j_username, password=neo4j_password)
+
+# remove all nodes
+graph.query("MATCH (n) DETACH DELETE n")
+
+# ingest
+import_query = json.load(open("data/microservices.json", "r"))["query"]
+graph.query(import_query)
+print("Total nodes: ", graph.query("MATCH (n) RETURN count(n)"))
+print("Total edges: ", graph.query("MATCH ()-->() RETURN count(*)"))