-
-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test(autofix): Add more vcr-based tests for autofix scenarios #1681
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
0dec62d
root cause question asking tests + ci flag check
jennmueng e83a218
rethink + coding by itself
jennmueng 7605cf6
test_check_and_mark_recent_autofix_runs with real db runs
jennmueng 87e7ff2
test update code change
jennmueng ea90a2d
lru cache
jennmueng 17149e2
precommit
jennmueng f3ed387
encrypted cassettes
jennmueng 27226a8
encrypted cassettes 2
jennmueng d7de6ae
encrypted cassettes 3
jennmueng 8cdce85
encrypted cassettes 4
jennmueng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,4 +1,4 @@ | ||
from typing import Any, Optional | ||
from typing import Any | ||
|
||
import sentry_sdk | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/agent/_encrypted_cassettes/test_gemini_generate_structured.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
-1 Byte
(100%)
tests/automation/agent/_encrypted_cassettes/test_gemini_generate_text.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/agent/_encrypted_cassettes/test_gemini_generate_text_stream.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...ion/agent/_encrypted_cassettes/test_gemini_generate_text_stream_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...automation/agent/_encrypted_cassettes/test_gemini_generate_text_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/autofix/_encrypted_cassettes/test_anthropic_generate_text.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...utomation/autofix/_encrypted_cassettes/test_anthropic_generate_text_stream.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...utofix/_encrypted_cassettes/test_anthropic_generate_text_stream_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...ation/autofix/_encrypted_cassettes/test_anthropic_generate_text_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
-1.17 KB
(99%)
tests/automation/autofix/_encrypted_cassettes/test_autofix_create_pr.yaml.encrypted
Binary file not shown.
Binary file added
BIN
+60.8 KB
...autofix/_encrypted_cassettes/test_autofix_restart_from_point_with_feedback.yaml.encrypted
Binary file not shown.
Binary file added
BIN
+195 KB
tests/automation/autofix/_encrypted_cassettes/test_autofix_run_coding.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+44.3 KB
(120%)
tests/automation/autofix/_encrypted_cassettes/test_autofix_run_full.yaml.encrypted
Binary file not shown.
Binary file added
BIN
+260 KB
...s/automation/autofix/_encrypted_cassettes/test_autofix_run_question_asking.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+27.6 KB
(130%)
...tomation/autofix/_encrypted_cassettes/test_autofix_run_root_cause_analysis.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+1 Byte
(100%)
tests/automation/autofix/_encrypted_cassettes/test_gemini_generate_structured.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/autofix/_encrypted_cassettes/test_gemini_generate_text.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...ion/autofix/_encrypted_cassettes/test_gemini_generate_text_from_web_search.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...s/automation/autofix/_encrypted_cassettes/test_gemini_generate_text_stream.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...n/autofix/_encrypted_cassettes/test_gemini_generate_text_stream_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...tomation/autofix/_encrypted_cassettes/test_gemini_generate_text_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...x/_encrypted_cassettes/test_get_completion_interrupts_with_queued_messages.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
-1 Byte
(100%)
tests/automation/autofix/_encrypted_cassettes/test_openai_generate_structured.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+1 Byte
(100%)
tests/automation/autofix/_encrypted_cassettes/test_openai_generate_text.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...s/automation/autofix/_encrypted_cassettes/test_openai_generate_text_stream.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...n/autofix/_encrypted_cassettes/test_openai_generate_text_stream_with_tools.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
...tomation/autofix/_encrypted_cassettes/test_openai_generate_text_with_tools.yaml.encrypted
Binary file not shown.
Binary file added
BIN
+122 KB
...tomation/autofix/_encrypted_cassettes/test_restart_step_with_user_response.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/autofix/_encrypted_cassettes/test_run_iteration.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
.../automation/autofix/_encrypted_cassettes/test_run_max_iterations_exception.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/autofix/_encrypted_cassettes/test_run_with_initial_prompt.yaml.encrypted
Binary file not shown.
Binary file modified
BIN
+0 Bytes
(100%)
tests/automation/autofix/_encrypted_cassettes/test_run_with_tool_calls.yaml.encrypted
Binary file not shown.
3,176 changes: 3,176 additions & 0 deletions
3,176
tests/automation/autofix/cassettes/test_autofix_restart_from_point_with_feedback.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
6,154 changes: 6,154 additions & 0 deletions
6,154
tests/automation/autofix/cassettes/test_autofix_run_coding.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
1,552 changes: 1,552 additions & 0 deletions
1,552
tests/automation/autofix/cassettes/test_autofix_run_question_asking.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
1,994 changes: 1,994 additions & 0 deletions
1,994
tests/automation/autofix/cassettes/test_restart_step_with_user_response.yaml
Large diffs are not rendered by default.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@functools.cache
was breaking vcr test cases as it was maintaining the cache across multiple test cases rather than cleaning itThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is cache clearing not feasible instead? This sounds like it would cause more repo_client initialization calls and add lag back in some cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lru cache in prod in this use case should work exactly the same as the existing cache with a limit instead of being unbounded. we can add cache clearing but we'll need to explicitly define that for test cases, and for the sake of unit testing the expected behavior is each test case is independent anyways so this removes an unexpected behavior
the limit of 8 was set instead of an infinite size as I don't foresee there being more than 8 repos in an autofix run
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking about the entire Python process vs. decorated function is redefined. In Autofix's context, when do new processes start? And when is the function redefined? Are they the same or different?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in a prod environment it should be equivalent as the function/class is defined when the process starts, the only difference is in testing environments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so in autofix prod, each celery process has its own cache for either the lru cache or cach method and they fundamentally would work the same
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok i trust u