autoedits e2e tests #6425

hitesh-1997 · 2024-12-19T18:29:02Z

Context

End to end testing setup for the auto-edits
https://playwright.dev/docs/test-snapshots is used to detect the differences in the rendering for the auto-edit diff view
Closes https://linear.app/sourcegraph/issue/CODY-4570/implement-a-manual-trigger-command

Test plan

Added unit test and e2e tests

valerybugakov · 2024-12-26T04:43:41Z

vscode/src/autoedits/autoedits-provider.ts

@@ -120,6 +124,14 @@ export class AutoeditsProvider implements vscode.InlineCompletionItemProvider, v
            vscode.window.onDidChangeTextEditorSelection(this.onSelectionChangeDebounced),
            vscode.workspace.onDidChangeTextDocument(event => {
                this.onDidChangeTextDocument(event)
+            }),
+            // Command used to trigger autoedits manually via command palette and is used by e2e test
+            vscode.commands.registerCommand('cody.command.autoedits-manual-trigger', async () => {


Let's register this command in the registerAutoEdits function body.

valerybugakov · 2024-12-26T04:46:04Z

vscode/src/autoedits/autoedits-provider.ts

+            }),
+            // Command used to trigger autoedits manually via command palette and is used by e2e test
+            vscode.commands.registerCommand('cody.command.autoedits-manual-trigger', async () => {
+                this.showAutoEdit(


What do you think about using an approach similar to autocomplete? This would invoke the provideInlineCompletionItems method with appropriate parameters.

public async manuallyTriggerCompletion(): Promise<void> { await vscode.commands.executeCommand('editor.action.inlineSuggest.hide') this.lastManualCompletionTimestamp = Date.now() await vscode.commands.executeCommand('editor.action.inlineSuggest.trigger') }

nice !!
This is much better, changed to the proposed implementation

valerybugakov · 2024-12-26T04:48:16Z

vscode/test/e2e/auto-edits.test.ts

+ *
+ * Currently these tests run only on macOS due to cross-platform rendering inconsistencies.
+ *
+ * Previous attempts and challenges encountered:


Thank you 🤩

valerybugakov · 2024-12-26T05:18:59Z

vscode/test/fixtures/workspace/auto-edits/diff-view-example-1.py

+"""
+<<<<


To keep E2E tests closer to the real-world setup, we usually mock server responses in vscode/test/fixtures/mock-server.ts. Do you think we can do that for autoedits? If we rely on the /.api/completions/code endpoint for suggestions generation, it's already mocked in the mock server implementation.

This is the first approach I tried when implementing e2e tests. There are the painpoints in using the mock-server:

The current mock-server has a hardcoded list of responses, which we check on case to case basis ior the mock list in case of ./api/completions/code endpoint, based on the content on the prompt. This setup is not very extensible if we want to add say 10-12 different image types from cody-chat-eval.
I think the current setup has a downside that I doesn't do thought the adapter layer in e2e setup, but this gives us much more flexibility in adding new test cases and simply port over the examples from cody-chat-eval.

Another pain-point would be that, the response returned from the server is a rewrite of <code_to_rewrite>, so in the hardcoded list above, we need to extract the logic either manually or we need to use the same setup i.e. mock-renderer, so in any case we would use the existing mock-renderer setup to get the response and extend the mock response list.

I think the current adapters are covered with extensive unit tests and should be fine to skip it in the e2e tests and follow the mock responses. Do you think we should continue with the setup we have in the PR ?

Thanks for sharing the context! I agree; we can change it in a follow-up if needed.

Could you leave this context as a comment in the test file for posterity?

Sure, Added detailed comment in the test file

valerybugakov · 2024-12-26T05:20:26Z

vscode/test/e2e/auto-edits.test.ts

+        // Wait for the diff view to stabilize - required to reduce flakiness
+        await page.waitForTimeout(500)
+
+        await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {


valerybugakov · 2024-12-26T05:20:39Z

vscode/test/e2e/auto-edits.test.ts

+        await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
+            maxDiffPixelRatio: 0.02,
+            maxDiffPixels: 1000,
+            clip: clip,


Suggested change

clip: clip,

clip,

valerybugakov

This is awesome! Thank you for implementing these tests.

## Context 1. Adding unit test case for the renderer manager 2. Build on top of #6425 ## Test plan Added unit test

1. Adding unit test case for the renderer manager 2. Build on top of #6425 Added unit test

hitesh-1997 force-pushed the hitesh/autoedit-e2e branch from b4a4af8 to d1763d3 Compare December 20, 2024 17:16

hitesh-1997 marked this pull request as ready for review December 20, 2024 17:18

hitesh-1997 requested review from valerybugakov and umpox December 20, 2024 17:18

hitesh-1997 force-pushed the hitesh/autoedit-e2e branch 2 times, most recently from ae45dbd to b3583e4 Compare December 20, 2024 19:12

hitesh-1997 marked this pull request as draft December 24, 2024 11:37

hitesh-1997 mentioned this pull request Dec 24, 2024

Hitesh/renderer testing #6458

Merged

hitesh-1997 marked this pull request as ready for review December 25, 2024 17:00

valerybugakov reviewed Dec 26, 2024

View reviewed changes

valerybugakov approved these changes Dec 26, 2024

View reviewed changes

hitesh-1997 added a commit that referenced this pull request Dec 26, 2024

Hitesh/renderer testing (#6458)

477bf1a

## Context 1. Adding unit test case for the renderer manager 2. Build on top of #6425 ## Test plan Added unit test

hitesh-1997 added 18 commits December 26, 2024 17:03

autoedits e2e tests

bd86d56

close the explorer window to capture larger diff

df6016d

add image clipping option to reduce flakyness

b8d3ee9

only run autoedit test

69de904

add another folder

769d8c6

test with fixed fonts

fcb8d91

test with fixed fonts

abed053

try platform specific snapshots

61b8cf5

add linux snapshots

0a83d31

run the test 100 times to detect flakyness

5d04a63

test in zen mode without platform specific diffs

da9ed9a

hide custom bar

70bab5a

autoedit: e2e tests run only on macos

e4c9969

check if run on macos

6a1d47d

e2e on all

6364783

e2e on all

75b128a

e2e on all

d4fe137

e2e on all

1d3bbb0

hitesh-1997 and others added 6 commits December 26, 2024 17:03

address pr comments

7d02592

restore ci workflow

5dafa82

restore ci workflow

fcde36a

address pr comments

d533343

Hitesh/renderer testing (#6458)

4218f06

1. Adding unit test case for the renderer manager 2. Build on top of #6425 Added unit test

address pr comments

30ce79c

hitesh-1997 force-pushed the hitesh/autoedit-e2e branch from 477bf1a to 30ce79c Compare December 26, 2024 12:05

restore parameters for test helper

4cdb0b3

hitesh-1997 merged commit b06f903 into main Dec 26, 2024
20 of 21 checks passed

hitesh-1997 deleted the hitesh/autoedit-e2e branch December 26, 2024 12:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

autoedits e2e tests #6425

autoedits e2e tests #6425

hitesh-1997 commented Dec 19, 2024 •

edited by valerybugakov

Loading

valerybugakov Dec 26, 2024

hitesh-1997 Dec 26, 2024

valerybugakov Dec 26, 2024

hitesh-1997 Dec 26, 2024

valerybugakov Dec 26, 2024

valerybugakov Dec 26, 2024

hitesh-1997 Dec 26, 2024 •

edited

Loading

valerybugakov Dec 26, 2024

valerybugakov Dec 26, 2024

hitesh-1997 Dec 26, 2024

valerybugakov Dec 26, 2024

valerybugakov Dec 26, 2024

hitesh-1997 Dec 26, 2024

valerybugakov left a comment

		"""
		<<<<

autoedits e2e tests #6425

autoedits e2e tests #6425

Conversation

hitesh-1997 commented Dec 19, 2024 • edited by valerybugakov Loading

Context

Test plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hitesh-1997 Dec 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valerybugakov left a comment

Choose a reason for hiding this comment

hitesh-1997 commented Dec 19, 2024 •

edited by valerybugakov

Loading

hitesh-1997 Dec 26, 2024 •

edited

Loading