Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test(autofix): Add more vcr-based tests for autofix scenarios #1681

Merged
merged 10 commits into from
Jan 22, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/integrations/codecov/codecov_auth.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
import hashlib
import hmac
import json

import requests
from seer.automation.codegen.models import CodecovTaskRequest

from seer.configuration import AppConfig
from seer.dependency_injection import inject, injected
from werkzeug.exceptions import Unauthorized

OUTGOING_REQUEST_SIGNATURE_HEADER = "HTTP-X-GEN-AI-AUTH-SIGNATURE"

Expand Down
4 changes: 1 addition & 3 deletions src/seer/app.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,7 @@
from sentry_sdk.integrations.logging import LoggingIntegration
from werkzeug.exceptions import GatewayTimeout, InternalServerError

from integrations.codecov.codecov_auth import (
CodecovAuthentication,
)
from integrations.codecov.codecov_auth import CodecovAuthentication
from seer.anomaly_detection.models.external import (
DeleteAlertDataRequest,
DeleteAlertDataResponse,
Expand Down
2 changes: 1 addition & 1 deletion src/seer/automation/codebase/repo_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -193,7 +193,7 @@ def _extract_id_from_pr_url(pr_url: str):
return pr_id

@classmethod
@functools.cache
@functools.lru_cache(maxsize=8)
Copy link
Member Author

@jennmueng jennmueng Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@functools.cache was breaking vcr test cases as it was maintaining the cache across multiple test cases rather than cleaning it

  • @functools.cache maintains its cache for the entire lifetime of the Python process
  • @functools.lru_cache creates a new cache instance each time the decorated function is redefined, which typically happens for each test module

This means that lru_cache naturally provides better test isolation out of the box, even without explicit cache clearing. This is one of the less-documented but important differences between the two decorators.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is cache clearing not feasible instead? This sounds like it would cause more repo_client initialization calls and add lag back in some cases

Copy link
Member Author

@jennmueng jennmueng Jan 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lru cache in prod in this use case should work exactly the same as the existing cache with a limit instead of being unbounded. we can add cache clearing but we'll need to explicitly define that for test cases, and for the sake of unit testing the expected behavior is each test case is independent anyways so this removes an unexpected behavior

the limit of 8 was set instead of an infinite size as I don't foresee there being more than 8 repos in an autofix run

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking about the entire Python process vs. decorated function is redefined. In Autofix's context, when do new processes start? And when is the function redefined? Are they the same or different?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in a prod environment it should be equivalent as the function/class is defined when the process starts, the only difference is in testing environments

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so in autofix prod, each celery process has its own cache for either the lru cache or cach method and they fundamentally would work the same

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok i trust u

def from_repo_definition(cls, repo_def: RepoDefinition, type: RepoClientType):
if type == RepoClientType.WRITE:
return cls(*get_write_app_credentials(), repo_def)
Expand Down
2 changes: 1 addition & 1 deletion src/seer/automation/codegen/step.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from typing import Any, Optional
from typing import Any

import sentry_sdk

Expand Down
2 changes: 1 addition & 1 deletion src/seer/automation/pipeline.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import abc
import logging
import uuid
from typing import Any, Generic, Optional, TypeVar
from typing import Any, Generic, TypeVar

import sentry_sdk
from celery import Task, signature
Expand Down
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.

Large diffs are not rendered by default.

6,154 changes: 6,154 additions & 0 deletions tests/automation/autofix/cassettes/test_autofix_run_coding.yaml

Large diffs are not rendered by default.

1,552 changes: 1,552 additions & 0 deletions tests/automation/autofix/cassettes/test_autofix_run_question_asking.yaml

Large diffs are not rendered by default.

1,994 changes: 1,994 additions & 0 deletions tests/automation/autofix/cassettes/test_restart_step_with_user_response.yaml

Large diffs are not rendered by default.

Loading
Loading