feat: support placeholders in patches

[ErikBjare]

For some reason Claude suddenly decided to spit out tons of placeholders in patches. This should let us parse them.

[ellipsis-dev]

Your free trial has expired. To keep using Ellipsis, sign up at https://app.ellipsis.dev for $20/seat/month or reach us at [email protected]

[codecov-commenter]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
71	1	70	0

View the top 1 failed tests by shortest run time

tests.test_eval test_eval_cli

Stack Traces | 31.3s run time

@pytest.mark.slow
    def test_eval_cli():
        runner = CliRunner()
        test_set = ["hello"]
        result = runner.invoke(
            main,
            [
                *test_set,
            ],
        )
        assert result
        assert result.exit_code == 0
>       assert "correct file" in result.output
E       AssertionError: assert 'correct file' in '=== Running evals ===\n=== Completed test hello ===\n=== Completed test hello ===\n=== Completed test hello ===\n\n=== Finished ===\n\n\n\n=== Model Results ===\n\nResults for model: openai/gpt-4o\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\nResults for model: anthropic/claude-3-5-sonnet-20240620\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\nResults for model: openrouter/meta-llama/llama-3.1-405b-instruct\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\n\n=== Model Comparison ===\nModel                                          hello\n---------------------------------------------  --------\nopenai/gpt-4o                                  ❌ 0.00s\nanthropic/claude-3-5-sonnet-20240620           ❌ 0.00s\nopenrouter/meta-llama/llama-3.1-405b-instruct  ❌ 0.00s\n\nResults saved to .../gptme/eval_results/eval_results_20240909_204514.csv\n'
E        +  where '=== Running evals ===\n=== Completed test hello ===\n=== Completed test hello ===\n=== Completed test hello ===\n\n=== Finished ===\n\n\n\n=== Model Results ===\n\nResults for model: openai/gpt-4o\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\nResults for model: anthropic/claude-3-5-sonnet-20240620\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\nResults for model: openrouter/meta-llama/llama-3.1-405b-instruct\nCompleted 1 tests in 0.00s:\n- hello in 0.00s (gen: 0.00s, run: 0.00s, eval: 0.00s)\n\n\n=== Model Comparison ===\nModel                                          hello\n---------------------------------------------  --------\nopenai/gpt-4o                                  ❌ 0.00s\nanthropic/claude-3-5-sonnet-20240620           ❌ 0.00s\nopenrouter/meta-llama/llama-3.1-405b-instruct  ❌ 0.00s\n\nResults saved to .../gptme/eval_results/eval_results_20240909_204514.csv\n' = <Result okay>.output

tests/test_eval.py:20: AssertionError

To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard

[ErikBjare]

Another note to add here: this essentially breaks the ability to edit code containing such comments.