diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index e902bb73be..d3c37bdccc 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -54,6 +54,7 @@ jobs: uses: astral-sh/ruff-action@v2 with: version: ">=0.8.2" + args: "check --ignore E402" - name: Build ragflow:nightly-slim run: | diff --git a/README.md b/README.md index ed3681da90..fd533431fb 100644 --- a/README.md +++ b/README.md @@ -305,7 +305,6 @@ docker build -f Dockerfile -t infiniflow/ragflow:nightly . ``` 127.0.0.1 es01 infinity mysql minio redis ``` - In **docker/service_conf.yaml.template**, update mysql port to `5455` and es port to `1200`, as specified in **docker/.env**. 4. If you cannot access HuggingFace, set the `HF_ENDPOINT` environment variable to use a mirror site: diff --git a/README_id.md b/README_id.md index a48a5005e4..54e21040fb 100644 --- a/README_id.md +++ b/README_id.md @@ -206,7 +206,7 @@ Coba demo kami di [https://demo.ragflow.io](https://demo.ragflow.io). 5. Buka browser web Anda, masukkan alamat IP server Anda, dan login ke RAGFlow. > Dengan pengaturan default, Anda hanya perlu memasukkan `http://IP_DEVICE_ANDA` (**tanpa** nomor port) karena port HTTP default `80` bisa dihilangkan saat menggunakan konfigurasi default. -6. Dalam [service_conf.yaml](./docker/service_conf.yaml), pilih LLM factory yang diinginkan di `user_default_llm` dan perbarui +6. Dalam [service_conf.yaml.template](./docker/service_conf.yaml.template), pilih LLM factory yang diinginkan di `user_default_llm` dan perbarui bidang `API_KEY` dengan kunci API yang sesuai. > Lihat [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) untuk informasi lebih lanjut. @@ -219,16 +219,9 @@ Untuk konfigurasi sistem, Anda perlu mengelola file-file berikut: - [.env](./docker/.env): Menyimpan pengaturan dasar sistem, seperti `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, dan `MINIO_PASSWORD`. -- [service_conf.yaml](./docker/service_conf.yaml): Mengonfigurasi aplikasi backend. +- [service_conf.yaml.template](./docker/service_conf.yaml.template): Mengonfigurasi aplikasi backend. - [docker-compose.yml](./docker/docker-compose.yml): Sistem ini bergantung pada [docker-compose.yml](./docker/docker-compose.yml) untuk memulai. -Anda harus memastikan bahwa perubahan pada file [.env](./docker/.env) sesuai dengan yang ada di file [service_conf.yaml](./docker/service_conf.yaml). - -> File [./docker/README](./docker/README.md) menyediakan penjelasan detail tentang pengaturan lingkungan dan konfigurasi aplikasi, -> dan Anda DIWAJIBKAN memastikan bahwa semua pengaturan lingkungan yang tercantum di -> [./docker/README](./docker/README.md) selaras dengan konfigurasi yang sesuai di -> [service_conf.yaml](./docker/service_conf.yaml). - Untuk memperbarui port HTTP default (80), buka [docker-compose.yml](./docker/docker-compose.yml) dan ubah `80:80` menjadi `:80`. @@ -278,11 +271,10 @@ docker build -f Dockerfile -t infiniflow/ragflow:nightly . docker compose -f docker/docker-compose-base.yml up -d ``` - Tambahkan baris berikut ke `/etc/hosts` untuk memetakan semua host yang ditentukan di **docker/service_conf.yaml** ke `127.0.0.1`: + Tambahkan baris berikut ke `/etc/hosts` untuk memetakan semua host yang ditentukan di **conf/service_conf.yaml** ke `127.0.0.1`: ``` 127.0.0.1 es01 infinity mysql minio redis ``` - Di **docker/service_conf.yaml**, perbarui port mysql ke `5455` dan es ke `1200`, sesuai dengan yang ditentukan di **docker/.env**. 4. Jika Anda tidak dapat mengakses HuggingFace, atur variabel lingkungan `HF_ENDPOINT` untuk menggunakan situs mirror: diff --git a/README_ja.md b/README_ja.md index 94369c95ce..62977d19b2 100644 --- a/README_ja.md +++ b/README_ja.md @@ -182,7 +182,7 @@ 5. ウェブブラウザで、プロンプトに従ってサーバーの IP アドレスを入力し、RAGFlow にログインします。 > デフォルトの設定を使用する場合、デフォルトの HTTP サービングポート `80` は省略できるので、与えられたシナリオでは、`http://IP_OF_YOUR_MACHINE`(ポート番号は省略)だけを入力すればよい。 -6. [service_conf.yaml](./docker/service_conf.yaml) で、`user_default_llm` で希望の LLM ファクトリを選択し、`API_KEY` フィールドを対応する API キーで更新する。 +6. [service_conf.yaml.template](./docker/service_conf.yaml.template) で、`user_default_llm` で希望の LLM ファクトリを選択し、`API_KEY` フィールドを対応する API キーで更新する。 > 詳しくは [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup) を参照してください。 @@ -193,12 +193,12 @@ システムコンフィグに関しては、以下のファイルを管理する必要がある: - [.env](./docker/.env): `SVR_HTTP_PORT`、`MYSQL_PASSWORD`、`MINIO_PASSWORD` などのシステムの基本設定を保持する。 -- [service_conf.yaml](./docker/service_conf.yaml): バックエンドのサービスを設定します。 +- [service_conf.yaml.template](./docker/service_conf.yaml.template): バックエンドのサービスを設定します。 - [docker-compose.yml](./docker/docker-compose.yml): システムの起動は [docker-compose.yml](./docker/docker-compose.yml) に依存している。 -[.env](./docker/.env) ファイルの変更が [service_conf.yaml](./docker/service_conf.yaml) ファイルの内容と一致していることを確認する必要があります。 +[.env](./docker/.env) ファイルの変更が [service_conf.yaml.template](./docker/service_conf.yaml.template) ファイルの内容と一致していることを確認する必要があります。 -> [./docker/README](./docker/README.md) ファイルは環境設定とサービスコンフィグの詳細な説明を提供し、[./docker/README](./docker/README.md) ファイルに記載されている全ての環境設定が [service_conf.yaml](./docker/service_conf.yaml) ファイルの対応するコンフィグと一致していることを確認することが義務付けられています。 +> [./docker/README](./docker/README.md) ファイルは環境設定とサービスコンフィグの詳細な説明を提供し、[./docker/README](./docker/README.md) ファイルに記載されている全ての環境設定が [service_conf.yaml.template](./docker/service_conf.yaml.template) ファイルの対応するコンフィグと一致していることを確認することが義務付けられています。 デフォルトの HTTP サービングポート(80)を更新するには、[docker-compose.yml](./docker/docker-compose.yml) にアクセスして、`80:80` を `:80` に変更します。 @@ -265,11 +265,10 @@ docker build -f Dockerfile -t infiniflow/ragflow:nightly . docker compose -f docker/docker-compose-base.yml up -d ``` - `/etc/hosts` に以下の行を追加して、**docker/service_conf.yaml** に指定されたすべてのホストを `127.0.0.1` に解決します: + `/etc/hosts` に以下の行を追加して、**conf/service_conf.yaml** に指定されたすべてのホストを `127.0.0.1` に解決します: ``` 127.0.0.1 es01 infinity mysql minio redis ``` - **docker/service_conf.yaml** で mysql のポートを `5455` に、es のポートを `1200` に更新します(**docker/.env** に指定された通り). 4. HuggingFace にアクセスできない場合は、`HF_ENDPOINT` 環境変数を設定してミラーサイトを使用してください: diff --git a/README_ko.md b/README_ko.md index d3769a9a19..16fe4f499d 100644 --- a/README_ko.md +++ b/README_ko.md @@ -189,7 +189,7 @@ 5. 웹 브라우저에 서버의 IP 주소를 입력하고 RAGFlow에 로그인하세요. > 기본 설정을 사용할 경우, `http://IP_OF_YOUR_MACHINE`만 입력하면 됩니다 (포트 번호는 제외). 기본 HTTP 서비스 포트 `80`은 기본 구성으로 사용할 때 생략할 수 있습니다. -6. [service_conf.yaml](./docker/service_conf.yaml) 파일에서 원하는 LLM 팩토리를 `user_default_llm`에 선택하고, `API_KEY` 필드를 해당 API 키로 업데이트하세요. +6. [service_conf.yaml.template](./docker/service_conf.yaml.template) 파일에서 원하는 LLM 팩토리를 `user_default_llm`에 선택하고, `API_KEY` 필드를 해당 API 키로 업데이트하세요. > 자세한 내용은 [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup)를 참조하세요. _이제 쇼가 시작됩니다!_ @@ -199,12 +199,12 @@ 시스템 설정과 관련하여 다음 파일들을 관리해야 합니다: - [.env](./docker/.env): `SVR_HTTP_PORT`, `MYSQL_PASSWORD`, `MINIO_PASSWORD`와 같은 시스템의 기본 설정을 포함합니다. -- [service_conf.yaml](./docker/service_conf.yaml): 백엔드 서비스를 구성합니다. +- [service_conf.yaml.template](./docker/service_conf.yaml.template): 백엔드 서비스를 구성합니다. - [docker-compose.yml](./docker/docker-compose.yml): 시스템은 [docker-compose.yml](./docker/docker-compose.yml)을 사용하여 시작됩니다. -[.env](./docker/.env) 파일의 변경 사항이 [service_conf.yaml](./docker/service_conf.yaml) 파일의 내용과 일치하도록 해야 합니다. +[.env](./docker/.env) 파일의 변경 사항이 [service_conf.yaml.template](./docker/service_conf.yaml.template) 파일의 내용과 일치하도록 해야 합니다. -> [./docker/README](./docker/README.md) 파일에는 환경 설정과 서비스 구성에 대한 자세한 설명이 있으며, [./docker/README](./docker/README.md) 파일에 나열된 모든 환경 설정이 [service_conf.yaml](./docker/service_conf.yaml) 파일의 해당 구성과 일치하도록 해야 합니다. +> [./docker/README](./docker/README.md) 파일에는 환경 설정과 서비스 구성에 대한 자세한 설명이 있으며, [./docker/README](./docker/README.md) 파일에 나열된 모든 환경 설정이 [service_conf.yaml.template](./docker/service_conf.yaml.template) 파일의 해당 구성과 일치하도록 해야 합니다. 기본 HTTP 서비스 포트(80)를 업데이트하려면 [docker-compose.yml](./docker/docker-compose.yml) 파일에서 `80:80`을 `:80`으로 변경하세요. @@ -269,11 +269,10 @@ docker build -f Dockerfile -t infiniflow/ragflow:nightly . docker compose -f docker/docker-compose-base.yml up -d ``` - `/etc/hosts` 에 다음 줄을 추가하여 **docker/service_conf.yaml** 에 지정된 모든 호스트를 `127.0.0.1` 로 해결합니다: + `/etc/hosts` 에 다음 줄을 추가하여 **conf/service_conf.yaml** 에 지정된 모든 호스트를 `127.0.0.1` 로 해결합니다: ``` 127.0.0.1 es01 infinity mysql minio redis ``` - **docker/service_conf.yaml** 에서 mysql 포트를 `5455` 로, es 포트를 `1200` 으로 업데이트합니다( **docker/.env** 에 지정된 대로). 4. HuggingFace에 접근할 수 없는 경우, `HF_ENDPOINT` 환경 변수를 설정하여 미러 사이트를 사용하세요: diff --git a/README_zh.md b/README_zh.md index 0d9e7ddc9a..1b756ae35f 100644 --- a/README_zh.md +++ b/README_zh.md @@ -183,7 +183,7 @@ 5. 在你的浏览器中输入你的服务器对应的 IP 地址并登录 RAGFlow。 > 上面这个例子中,您只需输入 http://IP_OF_YOUR_MACHINE 即可:未改动过配置则无需输入端口(默认的 HTTP 服务端口 80)。 -6. 在 [service_conf.yaml](./docker/service_conf.yaml) 文件的 `user_default_llm` 栏配置 LLM factory,并在 `API_KEY` 栏填写和你选择的大模型相对应的 API key。 +6. 在 [service_conf.yaml.template](./docker/service_conf.yaml.template) 文件的 `user_default_llm` 栏配置 LLM factory,并在 `API_KEY` 栏填写和你选择的大模型相对应的 API key。 > 详见 [llm_api_key_setup](https://ragflow.io/docs/dev/llm_api_key_setup)。 @@ -194,14 +194,14 @@ 系统配置涉及以下三份文件: - [.env](./docker/.env):存放一些基本的系统环境变量,比如 `SVR_HTTP_PORT`、`MYSQL_PASSWORD`、`MINIO_PASSWORD` 等。 -- [service_conf.yaml](./docker/service_conf.yaml):配置各类后台服务。 +- [service_conf.yaml.template](./docker/service_conf.yaml.template):配置各类后台服务。 - [docker-compose.yml](./docker/docker-compose.yml): 系统依赖该文件完成启动。 -请务必确保 [.env](./docker/.env) 文件中的变量设置与 [service_conf.yaml](./docker/service_conf.yaml) 文件中的配置保持一致! +请务必确保 [.env](./docker/.env) 文件中的变量设置与 [service_conf.yaml.template](./docker/service_conf.yaml.template) 文件中的配置保持一致! 如果不能访问镜像站点hub.docker.com或者模型站点huggingface.co,请按照[.env](./docker/.env)注释修改`RAGFLOW_IMAGE`和`HF_ENDPOINT`。 -> [./docker/README](./docker/README.md) 文件提供了环境变量设置和服务配置的详细信息。请**一定要**确保 [./docker/README](./docker/README.md) 文件当中列出来的环境变量的值与 [service_conf.yaml](./docker/service_conf.yaml) 文件当中的系统配置保持一致。 +> [./docker/README](./docker/README.md) 文件提供了环境变量设置和服务配置的详细信息。请**一定要**确保 [./docker/README](./docker/README.md) 文件当中列出来的环境变量的值与 [service_conf.yaml.template](./docker/service_conf.yaml.template) 文件当中的系统配置保持一致。 如需更新默认的 HTTP 服务端口(80), 可以在 [docker-compose.yml](./docker/docker-compose.yml) 文件中将配置 `80:80` 改为 `:80`。 @@ -275,11 +275,10 @@ docker build --build-arg NEED_MIRROR=1 -f Dockerfile -t infiniflow/ragflow:night docker compose -f docker/docker-compose-base.yml up -d ``` - 在 `/etc/hosts` 中添加以下代码,将 **docker/service_conf.yaml** 文件中的所有 host 地址都解析为 `127.0.0.1`: + 在 `/etc/hosts` 中添加以下代码,将 **conf/service_conf.yaml** 文件中的所有 host 地址都解析为 `127.0.0.1`: ``` 127.0.0.1 es01 infinity mysql minio redis ``` - 在文件 **docker/service_conf.yaml** 中,对照 **docker/.env** 的配置将 mysql 端口更新为 `5455`,es 端口更新为 `1200`。 4. 如果无法访问 HuggingFace,可以把环境变量 `HF_ENDPOINT` 设成相应的镜像站点: diff --git a/agent/README.md b/agent/README.md index 7f0a45f474..250149b84c 100644 --- a/agent/README.md +++ b/agent/README.md @@ -10,7 +10,7 @@ It is used to compose a complex work flow or agent. And this graph is beyond the DAG that we can use circles to describe our agent or work flow. Under this folder, we propose a test tool ./test/client.py which can test the DSLs such as json files in folder ./test/dsl_examples. Please use this client at the same folder you start RAGFlow. If it's run by Docker, please go into the container before running the client. -Otherwise, correct configurations in conf/service_conf.yaml is essential. +Otherwise, correct configurations in service_conf.yaml is essential. ```bash PYTHONPATH=path/to/ragflow python graph/test/client.py -h diff --git a/agent/README_zh.md b/agent/README_zh.md index 003ff4b7a4..411e31e2bf 100644 --- a/agent/README_zh.md +++ b/agent/README_zh.md @@ -11,7 +11,7 @@ 在这个文件夹下,我们提出了一个测试工具 ./test/client.py, 它可以测试像文件夹./test/dsl_examples下一样的DSL文件。 请在启动 RAGFlow 的同一文件夹中使用此客户端。如果它是通过 Docker 运行的,请在运行客户端之前进入容器。 -否则,正确配置 conf/service_conf.yaml 文件是必不可少的。 +否则,正确配置 service_conf.yaml 文件是必不可少的。 ```bash PYTHONPATH=path/to/ragflow python graph/test/client.py -h diff --git a/api/db/services/conversation_service.py b/api/db/services/conversation_service.py index 03092ece59..3cfa42a6b8 100644 --- a/api/db/services/conversation_service.py +++ b/api/db/services/conversation_service.py @@ -58,7 +58,7 @@ def structure_answer(conv, ans, message_id, session_id): chunk_list = [{ "id": chunk["chunk_id"], - "content": chunk["content"], + "content": chunk.get("content") if chunk.get("content") else chunk.get("content_with_content"), "document_id": chunk["doc_id"], "document_name": chunk["docnm_kwd"], "dataset_id": chunk["kb_id"], diff --git a/api/db/services/dialog_service.py b/api/db/services/dialog_service.py index 6fd4901872..36daadddac 100644 --- a/api/db/services/dialog_service.py +++ b/api/db/services/dialog_service.py @@ -195,7 +195,32 @@ def chat(dialog, messages, stream=True, **kwargs): dialog.vector_similarity_weight, doc_ids=attachments, top=dialog.top_k, aggs=False, rerank_mdl=rerank_mdl) - knowledges = [ck["content_with_weight"] for ck in kbinfos["chunks"]] + + # Group chunks by document ID + doc_chunks = {} + for ck in kbinfos["chunks"]: + doc_id = ck["doc_id"] + if doc_id not in doc_chunks: + doc_chunks[doc_id] = [] + doc_chunks[doc_id].append(ck["content_with_weight"]) + + # Create knowledges list with grouped chunks + knowledges = [] + for doc_id, chunks in doc_chunks.items(): + # Find the corresponding document name + doc_name = next((d["doc_name"] for d in kbinfos.get("doc_aggs", []) if d["doc_id"] == doc_id), doc_id) + + # Create a header for the document + doc_knowledge = f"Document: {doc_name} \nContains the following relevant fragments:\n" + + # Add numbered fragments + for i, chunk in enumerate(chunks, 1): + doc_knowledge += f"{i}. {chunk}\n" + + knowledges.append(doc_knowledge) + + + logging.debug( "{}->{}".format(" ".join(questions), "\n->".join(knowledges))) retrieval_tm = timer() @@ -592,12 +617,40 @@ def ask(question, kb_ids, tenant_id): knowledges = [ck["content_with_weight"] for ck in kbinfos["chunks"]] used_token_count = 0 + chunks_num = 0 for i, c in enumerate(knowledges): used_token_count += num_tokens_from_string(c) if max_tokens * 0.97 < used_token_count: knowledges = knowledges[:i] + chunks_num = chunks_num + 1 break + # Group chunks by document ID + doc_chunks = {} + counter_chunks = 0 + for ck in kbinfos["chunks"]: + if counter_chunks < chunks_num: + counter_chunks = counter_chunks + 1 + doc_id = ck["doc_id"] + if doc_id not in doc_chunks: + doc_chunks[doc_id] = [] + doc_chunks[doc_id].append(ck["content_with_weight"]) + + # Create knowledges list with grouped chunks + knowledges = [] + for doc_id, chunks in doc_chunks.items(): + # Find the corresponding document name + doc_name = next((d["doc_name"] for d in kbinfos.get("doc_aggs", []) if d["doc_id"] == doc_id), doc_id) + + # Create a header for the document + doc_knowledge = f"Document: {doc_name} \nContains the following relevant fragments:\n" + + # Add numbered fragments + for i, chunk in enumerate(chunks, 1): + doc_knowledge += f"{i}. {chunk}\n" + + knowledges.append(doc_knowledge) + prompt = """ Role: You're a smart assistant. Your name is Miss R. Task: Summarize the information from knowledge bases and answer user's question. diff --git a/api/ragflow_server.py b/api/ragflow_server.py index 713ba1881e..cd3bd8ccf4 100644 --- a/api/ragflow_server.py +++ b/api/ragflow_server.py @@ -18,10 +18,11 @@ # from beartype.claw import beartype_all # <-- you didn't sign up for this # beartype_all(conf=BeartypeConf(violation_type=UserWarning)) # <-- emit warnings from all code -import logging -import os from api.utils.log_utils import initRootLogger +initRootLogger("ragflow_server") +import logging +import os import signal import sys import time @@ -41,9 +42,6 @@ from api.utils import show_configs from rag.settings import print_rag_settings -LOG_LEVELS = os.environ.get("LOG_LEVELS", "") -initRootLogger("ragflow_server", LOG_LEVELS) - def update_progress(): while True: diff --git a/api/utils/log_utils.py b/api/utils/log_utils.py index 99654d20b6..c6df8e3456 100644 --- a/api/utils/log_utils.py +++ b/api/utils/log_utils.py @@ -28,7 +28,7 @@ def get_project_base_directory(): ) return PROJECT_BASE -def initRootLogger(logfile_basename: str, log_levels: str, log_format: str = "%(asctime)-15s %(levelname)-8s %(process)d %(message)s"): +def initRootLogger(logfile_basename: str, log_format: str = "%(asctime)-15s %(levelname)-8s %(process)d %(message)s"): logger = logging.getLogger() if logger.hasHandlers(): return @@ -48,8 +48,9 @@ def initRootLogger(logfile_basename: str, log_levels: str, log_format: str = "%( logging.captureWarnings(True) + LOG_LEVELS = os.environ.get("LOG_LEVELS", "") pkg_levels = {} - for pkg_name_level in log_levels.split(","): + for pkg_name_level in LOG_LEVELS.split(","): terms = pkg_name_level.split("=") if len(terms)!= 2: continue diff --git a/docs/configurations.md b/docs/configurations.md index 84c8388db8..1b0daaaab0 100644 --- a/docs/configurations.md +++ b/docs/configurations.md @@ -77,7 +77,7 @@ The [.env](https://github.com/infiniflow/ragflow/blob/main/docker/.env) file con - `MINIO_USER` The username for MinIO. - `MINIO_PASSWORD` - The password for MinIO. accordingly. + The password for MinIO. ### Redis diff --git a/docs/references/python_api_reference.md b/docs/references/python_api_reference.md index 4ca703f60e..fbf87fb892 100644 --- a/docs/references/python_api_reference.md +++ b/docs/references/python_api_reference.md @@ -1,4 +1,4 @@ -from sdk.python.ragflow_sdk import Agentfrom docutils.nodes import titlefrom sdk.python.ragflow_sdk import Agent--- +--- sidebar_position: 2 slug: /python_api_reference --- diff --git a/docs/release_notes.md b/docs/release_notes.md index 91f3a9e719..b59ebacb39 100644 --- a/docs/release_notes.md +++ b/docs/release_notes.md @@ -5,7 +5,25 @@ slug: /release_notes # Release notes -Key features and improvements in the latest releases. +Key features, improvements and bug fixes in the latest releases. + +## v0.14.1 + +Released on November 29, 2024. + +### Improvements + +Adds [Infinity's configuration file](https://github.com/infiniflow/ragflow/blob/main/docker/infinity_conf.toml) to facilitate integration and customization of Infinity as a document engine. From this release onwards, updates to Infinity's configuration can be made directly within RAGFlow and will take effect immediately after restarting RAGFlow using `docker compose`. [#3715](https://github.com/infiniflow/ragflow/pull/3715) + +### Fixed issues + +This release fixes the following issues: + +- Unable to display or edit content of a chunk after clicking it. +- A `'Not found'` error in Elasticsearch. +- Chinese text becoming garbled during parsing. +- A compatibility issue with Polars. +- A compatibility issue between Infinity and GraphRAG. ## v0.14.0 diff --git a/rag/nlp/__init__.py b/rag/nlp/__init__.py index 52687f276e..69573375a8 100644 --- a/rag/nlp/__init__.py +++ b/rag/nlp/__init__.py @@ -22,11 +22,11 @@ from . import rag_tokenizer import re import copy +import json import roman_numbers as r from word2number import w2n from cn2an import cn2an from PIL import Image -import json import chardet diff --git a/rag/svr/task_executor.py b/rag/svr/task_executor.py index c49ced4f1c..606bb07571 100644 --- a/rag/svr/task_executor.py +++ b/rag/svr/task_executor.py @@ -17,13 +17,14 @@ # from beartype.claw import beartype_all # <-- you didn't sign up for this # beartype_all(conf=BeartypeConf(violation_type=UserWarning)) # <-- emit warnings from all code -import logging import sys -import os - from api.utils.log_utils import initRootLogger +CONSUMER_NO = "0" if len(sys.argv) < 2 else sys.argv[1] +CONSUMER_NAME = "task_executor_" + CONSUMER_NO +initRootLogger(CONSUMER_NAME) - +import logging +import os from datetime import datetime import json import hashlib @@ -57,11 +58,6 @@ from rag.utils.redis_conn import REDIS_CONN, Payload from rag.utils.storage_factory import STORAGE_IMPL -CONSUMER_NO = "0" if len(sys.argv) < 2 else sys.argv[1] -CONSUMER_NAME = "task_executor_" + CONSUMER_NO -LOG_LEVELS = os.environ.get("LOG_LEVELS", "") -initRootLogger(CONSUMER_NAME, LOG_LEVELS) - BATCH_SIZE = 64 FACTORY = {