Allow for prompt caching to work by using `baibot_conversation_start_…

…time_utc` instead of `baibot_now_utc`

This patch introduces a new `baibot_conversation_start_time_utc`
variable which indicates the time the conversation got started.

Using `baibot_now_utc` is still possible, but given that the current
time is a moving target, its use is in conflict with prompt caching.

Because the new `baibot_conversation_start_time_utc` prompt variable
is a more reasonable default, we're now using it in all sample configs.

docs/configuration/text-generation.md

@@ -77,11 +77,14 @@ Prompts may contain the following **placeholder variables** which will be replac
 |---------------------------|-------------|---------|
 | `{{ baibot_name }}`       | Name of the bot as configured in the `user.name` field in the [Static configuration](./README.md#static-configuration) | `Baibot` |
 | `{{ baibot_model_id }}`   | Text-Generation model ID as configured in the [🤖 agent](../agents.md)'s configuration | `gpt-4o` |
-| `{{ baibot_now_utc }}`    | Current date and time in UTC | `2024-09-20 (Friday), 14:26:42 UTC` |
+| `{{ baibot_now_utc }}`    | Current date and time in UTC (⚠️ usage may break prompt caching - see below) | `2024-09-20 (Friday), 14:26:42 UTC` |
+| `{{ baibot_conversation_start_time_utc }}` | The date and time in UTC that the conversation started | `2024-09-20 (Friday), 14:26:42 UTC` |
+
+💡 `{{ baibot_now_utc }}` changes as time goes on, which prevents [prompt caching](https://platform.openai.com/docs/guides/prompt-caching) from working. It's better to use `{{ baibot_conversation_start_time_utc }}` in prompts, as its value doesn't change yet still orients the bot to the current date/time.
 
 Here's a prompt that combines some of the above variables:
 
-> You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+> You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
 
 
 ### 🌡️ Temperature Override

docs/sample-provider-configs/anthropic.yml

@@ -2,7 +2,7 @@ base_url: https://api.anthropic.com/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: claude-3-5-sonnet-20240620
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 8192
   max_context_tokens: 204800

docs/sample-provider-configs/groq.yml

@@ -2,7 +2,7 @@ base_url: https://api.groq.com/openai/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: llama3-70b-8192
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 131072

docs/sample-provider-configs/localai.yml

@@ -2,7 +2,7 @@ base_url: http://my-localai-self-hosted-service:8080/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: gpt-4
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/mistral.yml

@@ -2,7 +2,7 @@ base_url: https://api.mistral.ai/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: mistral-large-latest
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/ollama.yml

@@ -2,7 +2,7 @@ base_url: http://my-ollama-self-hosted-service:11434/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: gemma2:2b
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/openai-compatible.yml

@@ -2,7 +2,7 @@ base_url: ''
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: some-model
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 4096
   max_context_tokens: 128000

docs/sample-provider-configs/openai.yml

@@ -2,7 +2,7 @@ base_url: https://api.openai.com/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: gpt-4o
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 16384
   max_context_tokens: 128000

docs/sample-provider-configs/openrouter.yml

@@ -2,7 +2,7 @@ base_url: https://openrouter.ai/api/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: mattshumer/reflection-70b:free
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 2048
   max_context_tokens: 8192

docs/sample-provider-configs/together-ai.yml

@@ -2,7 +2,7 @@ base_url: https://api.together.xyz/v1
 api_key: YOUR_API_KEY_HERE
 text_generation:
   model_id: meta-llama/Meta-Llama-3.1-405B-Instruct-Turbo
-  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+  prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
   temperature: 1.0
   max_response_tokens: 2048
   max_context_tokens: 8192

etc/app/config.yml.dist

@@ -73,7 +73,7 @@ agents:
     #     api_key: ""
     #     text_generation:
     #       model_id: gpt-4o
-    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 16384
     #       max_context_tokens: 128000
@@ -97,7 +97,7 @@ agents:
     #     api_key: null
     #     text_generation:
     #       model_id: gpt-4
-    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 16384
     #       max_context_tokens: 128000
@@ -122,7 +122,7 @@ agents:
     #     api_key: null
     #     text_generation:
     #       model_id: "gemma2:2b"
-    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+    #       prompt: "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
     #       temperature: 1.0
     #       max_response_tokens: 4096
     #       max_context_tokens: 128000

src/agent/mod.rs

@@ -21,5 +21,5 @@ pub use provider::{AgentProvider, AgentProviderInfo, ControllerTrait};
 pub use purpose::AgentPurpose;
 
 pub(super) fn default_prompt() -> &'static str {
-    "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time now is: {{ baibot_now_utc }}."
+    "You are a brief, but helpful bot called {{ baibot_name }} powered by the {{ baibot_model_id }} model. The date/time of this conversation's start is: {{ baibot_conversation_start_time_utc }}."
 }

src/agent/provider/anthropic/controller.rs

@@ -72,6 +72,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -108,6 +109,7 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 

src/agent/provider/entity/text_generation/prompt_variables.rs

@@ -7,17 +7,33 @@ pub struct TextGenerationPromptVariables {
 
 impl Default for TextGenerationPromptVariables {
     fn default() -> Self {
-        Self::new("unnamed", "unknown-model", Utc::now())
+        let now = Utc::now();
+        Self::new("unnamed", "unknown-model", now, Some(now))
     }
 }
 
 impl TextGenerationPromptVariables {
-    pub fn new(bot_name: &str, model_id: &str, utc_time: DateTime<Utc>) -> Self {
+    pub fn new(
+        bot_name: &str,
+        model_id: &str,
+        now_time: DateTime<Utc>,
+        conversation_start_time: Option<DateTime<Utc>>,
+    ) -> Self {
         let mut map = HashMap::new();
 
         map.insert("baibot_name".to_string(), bot_name.to_string());
         map.insert("baibot_model_id".to_string(), model_id.to_string());
-        map.insert("baibot_now_utc".to_string(), format_utc_time(utc_time));
+        map.insert("baibot_now_utc".to_string(), format_utc_time(now_time));
+
+        let baibot_conversation_start_time_utc = match conversation_start_time {
+            Some(conversation_start_time) => format_utc_time(conversation_start_time),
+            None => "unknown".to_string(),
+        };
+
+        map.insert(
+            "baibot_conversation_start_time_utc".to_string(),
+            baibot_conversation_start_time_utc,
+        );
 
         Self { map }
     }
@@ -52,7 +68,18 @@ mod tests {
             .with_nanosecond(250000000)
             .unwrap();
 
-        let variables = TextGenerationPromptVariables::new("baibot", "gpt-4o", now_utc);
+        let conversation_start_time_utc = Utc
+            .with_ymd_and_hms(2024, 9, 19, 18, 34, 15)
+            .unwrap()
+            .with_nanosecond(250000000)
+            .unwrap();
+
+        let variables = TextGenerationPromptVariables::new(
+            "baibot",
+            "gpt-4o",
+            now_utc,
+            Some(conversation_start_time_utc),
+        );
 
         assert_eq!(
             variables.map.get("baibot_name"),
@@ -66,9 +93,13 @@ mod tests {
             variables.map.get("baibot_now_utc"),
             Some(&format_utc_time(now_utc))
         );
+        assert_eq!(
+            variables.map.get("baibot_conversation_start_time_utc"),
+            Some(&format_utc_time(conversation_start_time_utc))
+        );
 
-        let prompt = "Hello, I'm {{ baibot_name }} using {{ baibot_model_id }}. The date/time now is {{ baibot_now_utc }}.";
-        let expected = "Hello, I'm baibot using gpt-4o. The date/time now is 2024-09-20 (Friday), 18:34:15 UTC.";
+        let prompt = "Hello, I'm {{ baibot_name }} using {{ baibot_model_id }}. The date/time now is {{ baibot_now_utc }} and this conversation started at {{ baibot_conversation_start_time_utc }}.";
+        let expected = "Hello, I'm baibot using gpt-4o. The date/time now is 2024-09-20 (Friday), 18:34:15 UTC and this conversation started at 2024-09-19 (Thursday), 18:34:15 UTC.";
 
         assert_eq!(variables.format(prompt), expected);
     }

src/agent/provider/openai/controller.rs

@@ -63,6 +63,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -99,6 +100,7 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 

src/agent/provider/openai_compat/controller.rs

@@ -61,6 +61,7 @@ impl ControllerTrait for Controller {
         let messages = vec![LLMMessage {
             author: LLMAuthor::User,
             message_text: "Hello!".to_string(),
+            timestamp: chrono::Utc::now(),
         }];
 
         let conversation = LLMConversation { messages };
@@ -97,6 +98,7 @@ impl ControllerTrait for Controller {
             Some(LLMMessage {
                 author: LLMAuthor::Prompt,
                 message_text: prompt_text,
+                timestamp: chrono::Utc::now(),
             })
         };
 

src/controller/chat_completion/mod.rs

@@ -492,6 +492,7 @@ async fn handle_stage_text_generation(
             .text_generation_model_id()
             .unwrap_or("unknown-model".to_owned()),
         chrono::Utc::now(),
+        conversation.start_time(),
     );
 
     let params = TextGenerationParams {

src/controller/image/prompt.rs

@@ -46,6 +46,8 @@ mod tests {
 
     #[test]
     fn test_build_prompt() {
+        let timestamp = chrono::Utc::now();
+
         let test_cases = vec![
             // Simple case
             TestCase {
@@ -59,6 +61,7 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of a dog\nOther criteria:\n- Must be blue",
             },
@@ -68,14 +71,17 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::Assistant,
                     message_text: "Whatever".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "Must be 3-legged.\nMust be flying.".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of an elephant\nOther criteria:\n- Must be blue\n- Must be 3-legged.. Must be flying.",
             },
@@ -85,18 +91,22 @@ mod tests {
                 messages: vec![Message {
                     author: Author::User,
                     message_text: "Must be blue".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::Assistant,
                     message_text: "Whatever".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "Again".to_owned(),
+                    timestamp,
                 },
                 Message {
                     author: Author::User,
                     message_text: "again".to_owned(),
+                    timestamp,
                 }],
                 expected_prompt: "Generate a picture of a grizzly bear\nOther criteria:\n- Must be blue",
             },