feat(autoedit): Add more E2E test scenarios #25275
ci.yml
on: pull_request
fast_tests_matrix_prep
0s
slow_tests_matrix_prep
0s
build
2m 24s
Matrix: test-integration
Matrix: test-unit
Matrix: test-e2e
Annotations
10 errors, 11 warnings, and 1 notice
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded.
Failed worker ran 15 tests, last 10 tests were:
test/e2e/chat-context.test.ts:11:5 › chat followup context
test/e2e/chat-history.test.ts:17:3 › restore chat from sidebar history view
test/e2e/chat-history.test.ts:50:3 › delete chat from sidebar history view
test/e2e/chat-input.test.ts:25:3 › chat input focus
test/e2e/chat-input.test.ts:96:69 › chat toolbar and row UI
test/e2e/chat-input.test.ts:192:3 › chat model selector
test/e2e/chat-input.test.ts:243:3 › editing follow-up messages in chat view
test/e2e/chat-keyboard-shortcuts.test.ts:14:5 › chat keyboard shortcuts for sidebar chat
test/e2e/chat-keyboard-shortcuts.test.ts:35:5 › re-opening chat adds selection
test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
|
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded.
Failed worker ran 1 test:
test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
|
auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines:
vscode/test/e2e/auto-edits.test.ts#L154
1) auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines
Error: Screenshot comparison failed:
515 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 515 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 515 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:166:5
|
auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines:
vscode/test/e2e/auto-edits.test.ts#L154
1) auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: Screenshot comparison failed:
547 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 515 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 547 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:166:5
|
auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor:
vscode/test/e2e/auto-edits.test.ts#L154
2) auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor
Error: Screenshot comparison failed:
563 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-completion-29.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 553 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 563 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:180:5
|
auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor:
vscode/test/e2e/auto-edits.test.ts#L154
2) auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: Screenshot comparison failed:
563 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-completion-29.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 553 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 563 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:180:5
|
auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired:
vscode/test/e2e/auto-edits.test.ts#L154
3) auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position
Error: Screenshot comparison failed:
607 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 571 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 607 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:194:5
|
auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired:
vscode/test/e2e/auto-edits.test.ts#L154
3) auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: Screenshot comparison failed:
607 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 571 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 607 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:194:5
|
auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines:
vscode/test/e2e/auto-edits.test.ts#L154
4) auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines
Error: Screenshot comparison failed:
582 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 546 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 582 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:208:5
|
auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines:
vscode/test/e2e/auto-edits.test.ts#L154
4) auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines
Retry #1 ───────────────────────────────────────────────────────────────────────────────────────
Error: Screenshot comparison failed:
582 pixels (ratio 0.01 of all image pixels) are different.
Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png
Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-actual.png
Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-diff.png
Call log:
- expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png) with timeout 5000ms
- verifying given screenshot expectation
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- 546 pixels (ratio 0.01 of all image pixels) are different.
- waiting 100ms before taking screenshot
- taking page screenshot
- disabled all CSS animations
- waiting for fonts to load...
- fonts loaded
- captured a stable screenshot
- 582 pixels (ratio 0.01 of all image pixels) are different.
152 | await page.waitForTimeout(500)
153 |
> 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], {
| ^
155 | maxDiffPixelRatio: 0.02,
156 | maxDiffPixels: 500,
157 | clip,
at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28)
at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:208:5
|
fast_tests_matrix_prep
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
slow_tests_matrix_prep
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
test-integration (ubuntu)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
build
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
test-unit (ubuntu, 20)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
test-unit (ubuntu, 18)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
|
Slow Test:
vscode/chat-atFile.test.ts#L1
vscode/chat-atFile.test.ts took 1.6m
|
Slow Test:
vscode/initial-context.test.ts#L1
vscode/initial-context.test.ts took 1.4m
|
Slow Test:
vscode/chat-input.test.ts#L1
vscode/chat-input.test.ts took 1.2m
|
Slow Test:
vscode/command-custom.test.ts#L1
vscode/command-custom.test.ts took 1.2m
|
Slow Test:
vscode/command-edit.test.ts#L1
vscode/command-edit.test.ts took 53.1s
|
🎭 Playwright Run Summary
7 failed
auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines
auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor
auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position
auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines
auto-edits.test.ts:217:1 › autoedits: triggers inline decorations when multiple separate insertions are required on the same line
auto-edits.test.ts:239:1 › autoedits: does not show any suggestion if the suffix decoration spans further than the end of the file
chat-messages.test.ts:17:5 › chat assistant response code buttons ──────────────────────────────
4 skipped
46 passed (20.9m)
|
Artifacts
Produced during runtime
Name | Size | |
---|---|---|
playwright-recordings macos
|
13 MB |
|