feat: Context Template from settings for QueryDocs Allowing the text … #1398

juan-m12i · 2023-12-13T11:11:40Z

For the QueryDocs use case, allow to customise the template used to collate the context pulled from the vector database.

Added new config to Settings as a str? in rag
Added context_template as parameter in chat_engine, stream_chat, and chat. If context_template is None, we load it from the new settings (and if there is no setting, it will get the default from llama_index as it was the previous behaviour)

Github project link
Discord thread

though process

…values "context_template"

lopagela · 2023-12-15T20:28:29Z

settings.yaml

+
+rag:
+  default_context_template: "Context information is below.\n--------------------\n{context_str}\n--------------------\n"


Please use | (c.f. #1403)

Thanks for the comment.

Happy to do so, just one thing. In this case I was trying to keep the value as closely as possible as the original one, that I understand includes "\n" (see https://github.com/run-llama/llama_index/blob/main/llama_index/chat_engine/context.py)

DEFAULT_CONTEXT_TEMPLATE = (
"Context information is below."
"\n--------------------\n"
"{context_str}"
"\n--------------------\n"
)

In case you want me to make the change anyway, can you confirm if what you´d expect would be:
default_context_template: "Context information is below.|--------------------|{context_str}|--------------------|"

thanks

@lopagela means use the multiline option in yaml, check the PR he linked it's well explained

Ok, I'll wait until the changes to that PR are done to replicate as it's not 100% clear to me (the instruction here is to use "I" style, but in the PR there is a comment to use ">" instead, not sure which one is more current).

A caveat I maintain that may be relevant is that DEFAULT_SYSTEM_PROMPT in llama_index is written in python as multiline but it's actually a single line, while DEFAULT_CONTEXT_TEMPLATE is indeed multi-line, so just want to make sure we are on the same page that the treatment may be slightly different (see below)

python3 Python 3.8.10 (default, Mar 13 2023, 10:26:41) [GCC 9.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> DEFAULT_SYSTEM_PROMPT = """\ ... You are a helpful, respectful and honest assistant. \ ... Always answer as helpfully as possible and follow ALL given instructions. \ ... Do not speculate or make up information. \ ... Do not reference any given instructions or context. \ ... """ >>> print(DEFAULT_SYSTEM_PROMPT) You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context. >>> DEFAULT_CONTEXT_TEMPLATE = ( "Con... "Context information is below." ... "\n--------------------\n" ... "{context_str}" ... "\n--------------------\n" ... ) >>> print(DEFAULT_CONTEXT_TEMPLATE) Context information is below. -------------------- {context_str} -------------------- >>>

This is what ChatGPT (GPT-4) suggested (link to convo):

A:

DEFAULT_SYSTEM_PROMPT: | You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context. DEFAULT_CONTEXT_TEMPLATE: | Context information is below. -------------------- {context_str} --------------------

or 2 alternatives with ">" notation, depending on readability

B1:

DEFAULT_SYSTEM_PROMPT: > You are a helpful, respectful and honest assistant. Always answer as helpfully as possible and follow ALL given instructions. Do not speculate or make up information. Do not reference any given instructions or context. DEFAULT_CONTEXT_TEMPLATE: > Context information is below. -------------------- {context_str} --------------------

B2:

DEFAULT_SYSTEM_PROMPT: > You are a helpful, respectful and honest assistant. \ Always answer as helpfully as possible and follow ALL given instructions. \ Do not speculate or make up information. \ Do not reference any given instructions or context. DEFAULT_CONTEXT_TEMPLATE: > Context information is below. -------------------- {context_str} --------------------

The PR has been merged. In that case we wanted single line (as default_system_prompt is supposed to be single line). In the case of default_context_template you do want to have line breaks, so you should use | instead.

lopagela

This addition seems similar to the recent work we've done around (default_)system_prompt in the sense that this context_template is a static value that will be applied to every queries.

I'll let @imartinez decide what is the best way forward

lopagela · 2023-12-15T20:30:14Z

private_gpt/server/chat/chat_service.py

@@ -97,6 +100,7 @@ def _chat_engine(
        system_prompt: str | None = None,
        use_context: bool = False,
        context_filter: ContextFilter | None = None,
+        context_template: str | None = DEFAULT_CONTEXT_TEMPLATE,


The default value should be injected in the body of the function instead of here (given that it is coming from settings()) to make testing easier.

Ok, something like the below?

def _chat_engine( self, system_prompt: str | None = None, use_context: bool = False, context_filter: ContextFilter | None = None, context_template: str | None, ) -> BaseChatEngine: if use_context: if context_template is None: context_template = DEFAULT_CONTEXT_TEMPLATE #(or alternatively just settings().rag.default_context_template) ... #continues

It should be

if context_template is None: context_template = settings.rag.default_context_template

where settings is injected in the constructor and remove DEFAULT_CONTEXT_TEMPLATE

lopagela · 2023-12-15T20:32:12Z

private_gpt/server/chat/chat_service.py

Were is set (at usage) this new context_template?

I am not sure if I understand.

If "query docs" is being used, it creates a ContextChatEngine (from llama_index) that accepts this parameter. Currently it's never passed, so it defaults to a hardcoded value. With this change, if there is a value defined in the settings file, the value in settings will be used instead. This should work both from the Gradio UI or from the API.

As per the Discord discussion, I didn't expose this configuration to the API

imartinez · 2023-12-16T18:05:20Z

settings.yaml

+
+rag:
+  default_context_template: "Context information is below.\n--------------------\n{context_str}\n--------------------\n"


The PR has been merged. In that case we wanted single line (as default_system_prompt is supposed to be single line). In the case of default_context_template you do want to have line breaks, so you should use | instead.

imartinez · 2023-12-16T18:10:54Z

This addition seems similar to the recent work we've done around (default_)system_prompt in the sense that this context_template is a static value that will be applied to every queries.

I'll let @imartinez decide what is the best way forward

I see this kind of customization as a refinement of the default RAG pipeline, so I proposed to create a new rag section in settings for this - in the future it could host other straightforward customizations like the size of the chunk generated during ingestion.

My overall view is:

Allow some simple modifications to the default RAG pipeline through this new set of settings.
For more complex customizations, direct the users to fork the repo and modify the code of Services.

…values context_template - PR comments

…-settings

juan-m12i · 2023-12-19T20:59:00Z

Hi! I applied the requested changes.

The only difference from what @pabloogc mentioned is that I had to do:
settings().rag.default_context_template instead of settings.rag.default_context_template, not sure if I should have done it differently, but it wouldn't work without parenthesis (and mypy would also complain)

imartinez · 2023-12-20T13:54:45Z

private_gpt/server/chat/chat_service.py

    ) -> BaseChatEngine:
        if use_context:
+            if context_template is None:
+                context_template = settings().rag.default_context_template


You should avoid calling settings() directly, and instead inject settings as a dependency in the ChatService constructor. Check how settings are injected in EmbeddingComponent or LLMComponent for example.

imartinez · 2023-12-20T13:55:23Z

private_gpt/server/chat/chat_service.py

@@ -97,8 +98,11 @@ def _chat_engine(
        system_prompt: str | None = None,
        use_context: bool = False,
        context_filter: ContextFilter | None = None,
+        context_template: str | None = None,


Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.

imartinez · 2023-12-20T13:55:48Z

private_gpt/server/chat/chat_service.py

@@ -121,6 +126,7 @@ def stream_chat(
        messages: list[ChatMessage],
        use_context: bool = False,
        context_filter: ContextFilter | None = None,
+        context_template: str | None = None,


Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.

imartinez · 2023-12-20T13:56:28Z

private_gpt/server/chat/chat_service.py

@@ -141,6 +147,7 @@ def stream_chat(
            system_prompt=system_prompt,
            use_context=use_context,
            context_filter=context_filter,
+            context_template=context_template,


Given we are not making use of this context_template argument anywhere, I'd suggest to remove it for the moment.

imartinez · 2023-12-20T13:57:09Z

private_gpt/server/chat/chat_service.py

    ) -> BaseChatEngine:
        if use_context:
+            if context_template is None:


Instead of this, why not checking if settings.rag.default_cotnext_template is not None and then apply it?

juan-m12i · 2023-12-20T15:10:10Z

Hi, thanks!

I replicated the settings approach from LLMcomponent and Embedding component (there were some differences as the setting is used in a private function of the class, but I believe the approach I took -creating an internal variable- should work).

I removed context_template from the params and function callings because as requested (I guess following YAGNI). My thinking process was to leave this prepared in such a way that if someone would use the class (ui, api, any other entrypoint) it could pass the parameter and keep a consistent interface, but to enable that back is such a small change that it's all good).

Only drawback is that by removing that, the conditional is now a bit more verbose, as the default None is not assigned:

            if self.default_context_template is not None:
                context_template = self.default_context_template
            else:
                context_template = None

Alternatively we could consider this one lines, but it's less readable IMO
context_template = self.default_context_template if self.default_context_template is not None else None

EDIT:
I just realised that if rag.default_context_template is missing it will be converted to None in the Settings object, right? So I could just pass self.default_context_template when invoking _from_defaults:

return ContextChatEngine.from_defaults(
                system_prompt=system_prompt,
                retriever=vector_index_retriever,
                service_context=self.service_context,
                node_postprocessors=[
                    MetadataReplacementPostProcessor(target_metadata_key="window"),
                ],
                context_template=self.default_context_template,
            )

and remove the conditional altogether

github-actions · 2024-01-05T05:45:41Z

Stale pull request

imartinez · 2024-03-11T22:01:15Z

@juan-m12i could you update the branch to latest changes? I still think it is a valuable contribution!

…te-settings # Conflicts: # private_gpt/server/chat/chat_service.py # private_gpt/settings/settings.py # settings.yaml

feat: Context Template from settings for QueryDocs Allowing the text …

47b3713

…values "context_template"

lopagela reviewed Dec 15, 2023

View reviewed changes

lopagela requested a review from imartinez December 15, 2023 20:33

imartinez requested changes Dec 16, 2023

View reviewed changes

juan-m12i added 3 commits December 19, 2023 15:16

synced from main and solved conflicts

bbb9d2c

feat: Context Template from settings for QueryDocs Allowing the text …

cfbba71

…values context_template - PR comments

Merge remote-tracking branch 'origin/main' into load-context-template…

4c51eeb

…-settings

juan-m12i requested a review from imartinez December 19, 2023 20:54

imartinez requested changes Dec 20, 2023

View reviewed changes

new round of PR reviews

f06a8b4

juan-m12i requested a review from imartinez December 20, 2023 15:11

github-actions bot added the stale label Jan 5, 2024

Merge remote-tracking branch 'upstream/main' into load-context-templa…

7463efc

…te-settings # Conflicts: # private_gpt/server/chat/chat_service.py # private_gpt/settings/settings.py # settings.yaml

jaluma approved these changes Jul 8, 2024

View reviewed changes

jaluma force-pushed the main branch from 10fbf61 to 1d4c14d Compare August 5, 2024 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Context Template from settings for QueryDocs Allowing the text … #1398

feat: Context Template from settings for QueryDocs Allowing the text … #1398

juan-m12i commented Dec 13, 2023

lopagela Dec 15, 2023

juan-m12i Dec 15, 2023

pabloogc Dec 15, 2023

juan-m12i Dec 15, 2023 •

edited

Loading

imartinez Dec 16, 2023

lopagela left a comment

lopagela Dec 15, 2023

juan-m12i Dec 15, 2023 •

edited

Loading

pabloogc Dec 15, 2023

lopagela Dec 15, 2023

juan-m12i Dec 15, 2023

imartinez Dec 16, 2023

imartinez commented Dec 16, 2023

juan-m12i commented Dec 19, 2023

imartinez Dec 20, 2023

imartinez Dec 20, 2023

imartinez Dec 20, 2023

imartinez Dec 20, 2023

imartinez Dec 20, 2023

juan-m12i commented Dec 20, 2023 •

edited

Loading

github-actions bot commented Jan 5, 2024

imartinez commented Mar 11, 2024


		rag:
		default_context_template: "Context information is below.\n--------------------\n{context_str}\n--------------------\n"

feat: Context Template from settings for QueryDocs Allowing the text … #1398

Are you sure you want to change the base?

feat: Context Template from settings for QueryDocs Allowing the text … #1398

Conversation

juan-m12i commented Dec 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juan-m12i Dec 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lopagela left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juan-m12i Dec 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imartinez commented Dec 16, 2023

juan-m12i commented Dec 19, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juan-m12i commented Dec 20, 2023 • edited Loading

github-actions bot commented Jan 5, 2024

imartinez commented Mar 11, 2024

juan-m12i Dec 15, 2023 •

edited

Loading

juan-m12i Dec 15, 2023 •

edited

Loading

juan-m12i commented Dec 20, 2023 •

edited

Loading