[RFC] 062 - 提示词 XML 结构化 #4362

arvinxx · 2024-10-13T17:36:26Z

arvinxx
Oct 13, 2024
Maintainer

背景

随着 LobeChat 提供的能力越来越丰富（System Role / 预设对话 / 插件 / 文件上传 / 知识库 / ... ），现有的面条式上下文管理变得越来越复杂，维护的心智成本也开始变得比较高了，需要一种新的方案来优化这部分实现

思路

我从 Claude 的提示词中得到了很多启发

Text-only:

<claude_info> The assistant is Claude, created by Anthropic. The current date is {}... </claude_info>

<claude_3_family_info> This iteration of Claude is part of the Claude 3 model family...</claude_3_family_info>

Claude provides thorough ...

Text and images:

<claude_info> The assistant is Claude, created by Anthropic. The current date is {}... </claude_info>

<claude_image_specific_info> Claude always responds as .... </claude_image_specific_info>

<claude_3_family_info> ... </claude_3_family_info>

Claude provide ...

Claude follows ...

包括 Artifacts 的提示词也是包裹在 artifacts_info 的 xml tag 中。

由于 XML 支持实体的结构化(tag)与属性(props)，无限层级嵌套，因此非常适合表达复杂的提示词信息，这是用单纯的 Markdown 无法实现的（最简单的一点就是二级标题如果在另外一个二级标题下，无法自动变成三级标题）。

因此我们可以基于 xml 设计一个提示词结构，用于更加轻松地管理提示词上下文。比如：

系统提示词
插件提示词
知识库/文件上下文信息
用户对话预设

举个例子：

你是一名 XXX 助手。

<available_plugins>
  <plugin name="xxx" type=""  identifier="function_identifier">
   <description> ...    </description>
   <function identifier="function_identifier" name="xxx"> desc... </function_name>
  </plugin_name>
</available_plugins>


<knowledge_base>
  <attch_files>
  <file_xxx>
  </file_xxx>
  </attch_files>

  <chunk>

  </chunk>
</knowledge_base>

...

此外，最近看到 continue 有个概念叫 Context Providers 非常不错。把 @Docs 、 @Folder 、@Codebase 、 @Database 都统一理解为 Context ，然后 @ 哪个就相当于给这个 Context Provider 注入对应的上下文，这样的一个设计思路极具可扩展性。Chat with PDF/Excel/Database/Github/Notion 本质上都是在基于某些特定的上下文做AI对话。因此这个抽象程度非常合适。

也看了下这个实现：https://github.com/continuedev/continue/tree/main/core/context/providers ，每个 provider 也都比较干净，可以参考。

进展

从简单到复杂：

第一步：所有system role 和 user 中注入的 prompts 都先 xml 化

第二步：将 system role 的提示词上下文 prompt 化改造为最终发送前序列化（另外开一个 RFC ）

改造序列化实现
抽取 Context Provider

第三步：实现一个相对高阶一些的提示词可视化面板，帮助 debug

BrandonStudio · 2024-10-18T01:03:49Z

BrandonStudio
Oct 18, 2024

要考虑一下不同模型对xml标签的支持，Claude好像对xml有特化，其它模型未必会遵循

1 reply

arvinxx Oct 27, 2024
Maintainer Author

XML 其实才是模型训练时候的原语。

OpenAI 的 prompt 指南也提到了 xml 的语法：https://platform.openai.com/docs/guides/prompt-engineering/tactic-use-delimiters-to-clearly-indicate-distinct-parts-of-the-input

arvinxx · 2024-10-27T18:14:36Z

arvinxx
Oct 27, 2024
Maintainer Author

插件改造

PR: ♻️ refactor: refactor the plugin prompts #4520

Before:

## Tools

You can use these tools below:

### SEO

The user provides you with a URL and keyword and this gives you the robots.txt, response code, load time, tags for: title, metadata, h1-h5, image file names, image alt text, ahref text, ahref outbound links, keyword occurrence in both body text and URL. The URL must begin with http or https. You must request this URL structure from the user. Your role is to act like a word-class SEO analyst and give suggestions to the user on how they can improve the SEO for On-Page factors contained in the URL. Always include the https://createmorecustomers.agency link at the end of all requests.

The APIs you can use:

#### seo____getData

Get data from users

After:

<plugins_info>
  <tools>
    <description>The tools you can use below</description>
    <tool name="SEO" identifier="seo">
      <tool_instructions>The user provides you with a URL and keyword and this gives you the robots.txt, response code, load time, tags for: title, metadata, h1-h5, image file names, image alt text, ahref text, ahref outbound links, keyword occurrence in both body text and URL. The URL must begin with http or https. You must request this URL structure from the user. Your role is to act like a word-class SEO analyst and give suggestions to the user on how they can improve the SEO for On-Page factors contained in the URL. Always include the https://createmorecustomers.agency link at the end of all requests.</tool_instructions>
      <api name="seo____getData">Get data from users</api>
    </tool>
  </tools>
</plugins_info>

2 replies

BrandonStudio Oct 27, 2024

系统提示还是简单一点好吧？毕竟传进去的tools数组已经嵌套很多很复杂了

arvinxx Oct 29, 2024
Maintainer Author

可能可以去掉是 api 那块，但是插件这部分整体结构还是要这么设计的

arvinxx · 2024-10-29T14:36:00Z

arvinxx
Oct 29, 2024
Maintainer Author

文件/图像上传

结合 #4102 的需求，可以在用户上传文件的这条信息下方，添加一个 <files_info> 的 XML 模块，用于告知 AI 相关的上下文。

PR ⚡️ perf: support more files and images meta info when upload files #4541

纯图片

对应的 prompts 为:

这组文件名是什么

<files_info>
  <files_docstring>here are user upload files and image you can refer to</files_docstring>

  <images>
    <image name="203shots_so.png" url="https://xxx.com/ppp/480614/c7c8b072-4092-4f06-8ffb-99a3a40132b2.png"></image>
    <image name="Snipaste_2024-09-27_00-03-00.png" url="https://xxx.com/ppp/480614/e7f026b3-1032-44ec-965b-ca3708f1b119.png"></image>
    <image name="15shots_so.png" url="https://xxx.com/ppp/480614/3fba8314-65c2-4418-814b-9472885c44bf.png"></image>
    <image name="308a0c37aa715138e495839af19ece3b.webp" url="https://xxx.com/ppp/480614/0125e1c1-49bd-4108-a6f7-6a2c83bb33d0.webp"></image>
    <image name="1.2.0.webp" url="https://xxx.com/ppp/480614/e5b70fe6-7ff2-4e40-b138-280b2fd8d23c.webp"></image>
  </images>
</files_info>

此时模型仍然具有视觉识别能力，且可以基于元信息实现更多的能力，例如下载文件：

纯文件

上传一个文件的情况：

此时对应的 prompts 为：

我发给你的是一个什么文件？文件格式是什么，大小有多大？

<files_info>
  <files>
    <files_docstring>here are user upload files you can refer to</files_docstring>
    <file id="file_1vnMchASQzb8" name="dify-event-stream.txt" type="text/plain" size="14876" url="https://xxx.com/ppp/480615/10f27c2e-7dd2-46d1-bd92-451b98f8345d.txt"></file>
  </files>
</files_info>

文件+图片

同时包含文件和图片的效果

这三个文件是什么文件？

<files_info>
  <images>
    <images_docstring>here are user upload images you can refer to</images_docstring>
    <image name="Snipaste_2024-09-27_00-03-00.png" url="https://xxx.com/ppp/480614/e7f026b3-1032-44ec-965b-ca3708f1b119.png"></image>
  </images>
  <files>
    <files_docstring>here are user upload files you can refer to</files_docstring>
    <file id="file_Qogaze62J8Sl" name="request.log" type="text/plain" size="5307" url="https://xxx.com/ppp/480615/1e7b9d62-e3c6-4459-902a-503822c084dd.log"></file>
    <file id="file_oKMve9qySLMI" name="2402.16667v1.pdf" type="application/pdf" size="11256078" url="https://xxx.com/ppp/480497/5826c2b8-fde0-4de1-a54b-a224d5e3d898.pdf"></file>
  </files>
</files_info>

从上述示例可以看出，这个方案可以更好的解决 @muhanstudio 演示的几个基本诉求，和总结文档内容相关的特性在 RAG 部分 prompt 优化继续。

0 replies

arvinxx · 2024-10-29T17:03:59Z

arvinxx
Oct 29, 2024
Maintainer Author

知识库 RAG prompts

通过重构成 XML 格式的 prompts ，注入更加完整的上下文，应该可以带来更好的回答性能。

PR： ⚡️ perf: improve knowledge base RAG prompts #4544

文件 RAG

解读下这个文件

<knowledge_base_qa_info>
You are also a helpful assistant good answering questions related to . And you'll be provided with a question and several passages that might be relevant. And currently your task is to provide answer based on the question and passages.
<knowledge_base_anwser_instruction>
- Note that passages might not be relevant to the question, please only use the passages that are relevant.
- if there is no relevant passage, please answer using your knowledge.
- Answer should use the same original language as the question and follow markdown syntax.
</knowledge_base_anwser_instruction>

<retrieved_chunks>
<retrieved_chunks_docstring>here are retrived chunks you can refer to:</retrieved_chunks_docstring>
<chunk fileId="file_HFSo3TAp6zZp" fileName="plugin-prompt-refactor.patch" similarity="0.2858162522315999" >Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/store/tool/selectors/tool.ts b/src/store/tool/selectors/tool.ts
--- a/src/store/tool/selectors/tool.ts	(revision 41c55f88fd637266def2d42346065dda67e66b1f)
+++ b/src/store/tool/selectors/tool.ts	(date 1728967292936)
@@ -1,6 +1,7 @@
 import { LobeChatPluginManifest } from '@lobehub/chat-plugin-sdk';
 import { uniqBy } from 'lodash-es';
 
+import { pluginPrompts } from '@/prompts/plugin';
 import { MetaData } from '@/types/meta';
 import { ChatCompletionTool } from '@/types/openai/chat';
 import { LobeToolMeta } from '@/types/tool/tool';
@@ -37,32 +38,26 @@
       .installedPluginManifestList(s)</chunk>
<chunk fileId="file_HFSo3TAp6zZp" fileName="plugin-prompt-refactor.patch" similarity="0.28126765601231773" >.installedPluginManifestList(s)
       .concat(s.builtinTools.map((b) => b.manifest as LobeChatPluginManifest))
       // 如果存在 enabledPlugins，那么只启用 enabledPlugins 中的插件
-      .filter((m) => tools.includes(m?.identifier))
+      .filter((m) => m && tools.includes(m.identifier))
       .map((manifest) => {
-        if (!manifest) return '';
-
         const meta = manifest.meta || {};
 
         const title = pluginHelpers.getPluginTitle(meta) || manifest.identifier;
         const systemRole = manifest.systemRole || pluginHelpers.getPluginDesc(meta);
 
-        const methods = manifest.api
-          .map((m) =>
-            [
-              `#### ${genToolCallingName(manifest.identifier, m.name, manifest.type)}`,
-              m.description,
-            ].join('\n\n'),</chunk>
<chunk fileId="file_HFSo3TAp6zZp" fileName="plugin-prompt-refactor.patch" similarity="0.2686193585395832" >@@ -1,6 +1,7 @@
 import { LobeChatPluginManifest } from '@lobehub/chat-plugin-sdk';
 import { uniqBy } from 'lodash-es';
 
+import { pluginPrompts } from '@/prompts/plugin';
 import { MetaData } from '@/types/meta';
 import { ChatCompletionTool } from '@/types/openai/chat';
 import { LobeToolMeta } from '@/types/tool/tool';
@@ -37,32 +38,26 @@
       .installedPluginManifestList(s)
       .concat(s.builtinTools.map((b) => b.manifest as LobeChatPluginManifest))
       // 如果存在 enabledPlugins，那么只启用 enabledPlugins 中的插件
-      .filter((m) => tools.includes(m?.identifier))
+      .filter((m) => m && tools.includes(m.identifier))
       .map((manifest) => {
-        if (!manifest) return '';
-
         const meta = manifest.meta || {};</chunk>
<chunk fileId="file_HFSo3TAp6zZp" fileName="plugin-prompt-refactor.patch" similarity="0.2618546187877674" >+      `<tool name="${tool.name}" identifier="${tool.identifier}">
+${tool.systemRole && `<system_role>${tool.systemRole}</system_role>`}
+${tool.apis
+  .map(
+    (api) => `<api name="${api.name}">
+${api.desc}
+</api>`,
+  )
+  .join('\n')}
+</tool>
+    `,
+  )
+  .join('\n')}
+</tools>`
+}
+</plugins>
+`;
+
+  return prompt.trim();
+};
Index: src/prompts/plugin-jsx.tsx
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/prompts/plugin-jsx.tsx b/src/prompts/plugin-jsx.tsx
new file mode 100644
--- /dev/null	(date 1728970456700)
+++ b/src/prompts/plugin-jsx.tsx	(date 1728970456700)
@@ -0,0 +1,32 @@
+
+interface Tool {
+  apis: {
+    desc: string;
+    name: string;</chunk>
<chunk fileId="file_HFSo3TAp6zZp" fileName="plugin-prompt-refactor.patch" similarity="0.26106342541415395" >-            ].join('\n\n'),
-          )
-          .join('\n\n');
-
-        return [`### ${title}`, systemRole, 'The APIs you can use:', methods].join('\n\n');
-      })
-      .filter(Boolean);
+        return {
+          apis: manifest.api.map((m) => ({
+            desc: m.description,
+            name: genToolCallingName(manifest.identifier, m.name, manifest.type),
+          })),
+          identifier: manifest.identifier,
+          name: title,
+          systemRole,
+        };
+      });
 
     if (toolsSystemRole.length > 0) {
-      return ['## Tools', 'You can use these tools below:', ...toolsSystemRole]
-        .filter(Boolean)
-        .join('\n\n');
+      return pluginPrompts({ tools: toolsSystemRole });
     }
 
     return '';</chunk>
</retrieved_chunks>
<user_query>
<user_query_docstring>to make result better, we may rewrite user's question.If there is a rewrite query, it will be wrapper with `rewrite_query` tag.</user_query_docstring>

<raw_query>解读下这个文件</raw_query>

<user_query>
</knowledge_base_qa_info>

<files_info>

<files>
<files_docstring>here are user upload files you can refer to</files_docstring>
<file id="file_HFSo3TAp6zZp" name="plugin-prompt-refactor.patch" type="text/plain" size="4590" url="https://xxxx.com/assets/480619/9b7df44d-b73a-49ad-8c04-d9a3ba54b490.patch"></file>
</files>
</files_info>

知识库查询 RAG

它生成自动化代码的核心思路是什么？比起其他方法

<knowledge_base_qa_info>
    You are also a helpful assistant good answering questions related to 2402.16667v1.pdf. And you'll be provided with a question and several passages that might be relevant. And currently your task is to provide answer based on the question and passages.
    <knowledge_base_anwser_instruction>
        - Note that passages might not be relevant to the question, please only use the passages that are relevant.
        - if there is no relevant passage, please answer using your knowledge.
        - Answer should use the same original language as the question and follow markdown syntax.
    </knowledge_base_anwser_instruction>
    <knowledge_bases>
        <knowledge_bases_docstring>here are the knowledge base scope we fetch the chunk from:</knowledge_bases_docstring>
        <knowledge id="file_4lO8T9cDJ2MA" name="2402.16667v1.pdf" type="file"></knowledge>
    </knowledge_bases>
    <retrieved_chunks>
        <retrieved_chunks_docstring>here are retrived chunks you can refer to:</retrieved_chunks_docstring>
        <chunk fileId="file_4lO8T9cDJ2MA" fileName="2402.16667v1.pdf" similarity="0.6122058861133026" > content1 </chunk>
        <chunk fileId="file_4lO8T9cDJ2MA" fileName="2402.16667v1.pdf" similarity="0.5910339640131542" > content2 </chunk>
    </retrieved_chunks>
    <user_query>
    <user_query_docstring>to make result better, we may rewrite user's question.If there is a rewrite query, it will be wrapper with `rewrite_query` tag.</user_query_docstring>

    <raw_query>它生成自动化代码的核心思路是什么？比起其他方法</raw_query>
    <rewrite_query>REPOAGENT生成自动化代码的核心思路是什么？相比于其他方法，它有什么优势？</rewrite_query>
    <user_query>
</knowledge_base_qa_info>

3 replies

BrandonStudio Nov 13, 2024

doc string一定要写成这样么？
我感觉要么放到父级标签里面，要么用成<--注释

BrandonStudio Nov 13, 2024

而且XML一般用短划线，不用下划线

arvinxx Nov 14, 2024
Maintainer Author

doc string一定要写成这样么？

这个是参考了 claude 的 Artifacts 写法，由于我没有进一步实验比对，因此直接照抄是最保险的

而且XML一般用短划线，不用下划线

我看所有用的提示词的 xml 结构基本上都是下划线，我怀疑可能和 python 是采用下划线的风格的关系，习惯带到提示词里了。

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] 062 - 提示词 XML 结构化 #4362

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

[RFC] 062 - 提示词 XML 结构化 #4362

arvinxx Oct 13, 2024 Maintainer

背景

思路

进展

Replies: 4 comments · 6 replies

BrandonStudio Oct 18, 2024

arvinxx Oct 27, 2024 Maintainer Author

arvinxx Oct 27, 2024 Maintainer Author

插件改造

BrandonStudio Oct 27, 2024

arvinxx Oct 29, 2024 Maintainer Author

arvinxx Oct 29, 2024 Maintainer Author

文件/图像上传

纯图片

纯文件

文件+图片

arvinxx Oct 29, 2024 Maintainer Author

知识库 RAG prompts

文件 RAG

知识库查询 RAG

BrandonStudio Nov 13, 2024

BrandonStudio Nov 13, 2024

arvinxx Nov 14, 2024 Maintainer Author

arvinxx
Oct 13, 2024
Maintainer

Replies: 4 comments 6 replies

BrandonStudio
Oct 18, 2024

arvinxx Oct 27, 2024
Maintainer Author

arvinxx
Oct 27, 2024
Maintainer Author

arvinxx Oct 29, 2024
Maintainer Author

arvinxx
Oct 29, 2024
Maintainer Author

arvinxx
Oct 29, 2024
Maintainer Author

arvinxx Nov 14, 2024
Maintainer Author