Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade data_path for query_with_files #330

Merged
merged 6 commits into from
Nov 4, 2024

Conversation

JingofXin
Copy link
Contributor

@JingofXin JingofXin commented Oct 30, 2024

Upgrade details are as follows:

  1. Files are no longer transmitted with raw data encoding; instead, their paths are transmitted. The corresponding image or sound module is also responsible for decoding the data and saving it as a file.
  2. Two API functions are exposed for encoding and decoding requests with files:
    • Exposed APIs:
      • encode_query_with_filepaths(query: str = None, files: List[str] = None) -> str
      • decode_query_with_filepaths(query_files: str) -> Union[dict, str]
    • Encoding format: lazyllm-query{‘query’: 'xxxxxx', 'files':['path1', 'path2']}
  3. Enable bind to support merged encoding.
    lazyllm.TrainableModule('Mini-InternVL-Chat-2B-V1-5') | bind(_0, lazyllm_file='Describe this image')
  4. Expose 1 API function for merging requests:
    lazyllm_merge_query(*args: str) -> str
  5. A new passthrough method for file transfer is added: You can specify the module's ID, allowing only that module to receive the file.
    globals['lazyllm_files'][module._module_id] = ["path/to/file1"]
  6. Add FileFormatter, and it will support encoding and decoding queries with files by setting the formatter to either decode or encode.

LazyLLM-datapath-english

Assuming that the query and file have already been encoded, take them as input. Here are examples with sound and image, respectively:
image

lazyllm/common/bind.py Outdated Show resolved Hide resolved
lazyllm/module/module.py Outdated Show resolved Hide resolved
lazyllm/common/__init__.py Outdated Show resolved Hide resolved
lazyllm/common/common.py Outdated Show resolved Hide resolved
lazyllm/components/text_to_speech/utils.py Show resolved Hide resolved
lazyllm/engine/engine.py Outdated Show resolved Hide resolved
@wzh1994 wzh1994 merged commit 6359808 into LazyAGI:main Nov 4, 2024
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants