Add option to reduce context window

[drbarq]

This PR adds functionality to allow users to set a lower context window than their LLMs maximum context window size. This is useful when the models performance degrades significantly with larger context windows, allowing users to optimize the tradeoff between context length and model performance.

Background

With PR #4977 being merged in version 0.14, we now support dynamic context window sizes. This PR builds on that by allowing users to manually set a lower context window size than their LLMs maximum, which can be beneficial in cases where:

• The model shows performance degradation with very large contexts
• Users want to optimize for speed vs context length
• Memory or computational resources need to be conserved

Changes

• Add token count checking before adding new events to prevent exceeding the window
• Implement truncation logic when token count would exceed the configured limit
• Improve handling of first user message preservation to maintain conversation coherence
• Add comprehensive test case for context window parameter truncation

Configuration

Users can set max_input_tokens in two ways:

1. Through config.toml:

[llm]
max_input_tokens = 20000

2. Through environment variables:

export LLM_MAX_INPUT_TOKENS=20000

Implementation Details

• Token count is checked before adding each new event
• If adding an event would exceed the context window:
  1. The history is truncated using _apply_conversation_window
  2. Action-observation pairs are kept together
  3. The first user message is always preserved
• The truncation is done in a way that maintains conversation coherence

Testing

Added new test case test_context_window_parameter_truncation that verifies:

• Token count checking works correctly
• Truncation occurs when limit would be exceeded
• First user message is preserved
• Action-observation pairs stay together

This implements and enhances the changes from PR #5079.

[enyst]

Thank you for taking this up, @drbarq ! For the record, the reason for this PR "switch" is here.

[enyst]

I'm afraid this line doesn't really do what we want. We need to do the token counting on the messages that are sent to the LLM API. OpenHands works with events, and these are events, which it then 'translates' into messages and sends to the LLM API. You can see, as you try to make it work on your machine for your use case, that the number of tokens will not match, some events differ more than others but they all differ. 😅

An alternative is to define a custom exception like, let's say, TokenLimitExceeded, here, and move this check to the LLM class, then raise the exception when the token comparison fails. Then maybe the exception can be treated like this ContextWindowExceededError is. What do you think, does that make sense?

[drbarq]

ahh I see I think I understand. I was getting unpredictable behavior last night and that makes more sense and this definitely doesn't work. I'll take another crack at it when I get a chance today.

I do have a dumb question but I can't recreate it predictably. How is the config.toml file loaded? Is it only with certain commands? I can't tell if I broke something on my machine or I am assuming incorrect behavior

[enyst]

Let's figure it out! There is something a bit unpredictable about it: when you set values there, then run either with UI or with main.py or with cli.py, then config.toml is loaded. However, if running with UI, any settings that are defined in the UI (in the Settings window) will override the toml values.
Could that be what happens?

[enyst]

And it's absolutely a good question! We're still trying to document it properly, the simple thing that it turns out a bit difficult does say that something's wrong and we should rethink some of it.
For reference: #3220