Skip to content

feat(autoedit): Add more E2E test scenarios #25275

feat(autoedit): Add more E2E test scenarios

feat(autoedit): Add more E2E test scenarios #25275

Triggered via pull request January 10, 2025 09:54
Status Failure
Total duration 24m 6s
Artifacts 1

ci.yml

on: pull_request
fast_tests_matrix_prep
0s
fast_tests_matrix_prep
slow_tests_matrix_prep
0s
slow_tests_matrix_prep
Matrix: test-integration
Matrix: test-unit
Matrix: test-e2e
Fit to window
Zoom out
Zoom in

Annotations

10 errors, 11 warnings, and 1 notice
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded. Failed worker ran 15 tests, last 10 tests were: test/e2e/chat-context.test.ts:11:5 › chat followup context test/e2e/chat-history.test.ts:17:3 › restore chat from sidebar history view test/e2e/chat-history.test.ts:50:3 › delete chat from sidebar history view test/e2e/chat-input.test.ts:25:3 › chat input focus test/e2e/chat-input.test.ts:96:69 › chat toolbar and row UI test/e2e/chat-input.test.ts:192:3 › chat model selector test/e2e/chat-input.test.ts:243:3 › editing follow-up messages in chat view test/e2e/chat-keyboard-shortcuts.test.ts:14:5 › chat keyboard shortcuts for sidebar chat test/e2e/chat-keyboard-shortcuts.test.ts:35:5 › re-opening chat adds selection test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
test-e2e (macos)
Worker teardown timeout of 30000ms exceeded. Failed worker ran 1 test: test/e2e/chat-messages.test.ts:17:5 › chat assistant response code buttons
auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines: vscode/test/e2e/auto-edits.test.ts#L154
1) auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines Error: Screenshot comparison failed: 515 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines/macos/autoedits-suffix-decoration-70-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 515 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 515 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:166:5
auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines: vscode/test/e2e/auto-edits.test.ts#L154
1) auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 547 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-suffix-decoration-70.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-1c59f-edit-affects-existing-lines-retry1/macos/autoedits-suffix-decoration-70-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-suffix-decoration-70.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 515 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 547 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:166:5
auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor: vscode/test/e2e/auto-edits.test.ts#L154
2) auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor Error: Screenshot comparison failed: 563 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor/macos/autoedits-inline-completion-29-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-completion-29.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 553 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 563 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:180:5
auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor: vscode/test/e2e/auto-edits.test.ts#L154
2) auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 563 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-completion-29.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-912dc-mmediately-after-the-cursor-retry1/macos/autoedits-inline-completion-29-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-completion-29.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 553 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 563 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:180:5
auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired: vscode/test/e2e/auto-edits.test.ts#L154
3) auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position Error: Screenshot comparison failed: 607 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position/macos/autoedits-inline-decoration-insertion-30-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 571 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 607 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:194:5
auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired: vscode/test/e2e/auto-edits.test.ts#L154
3) auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 607 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-insertion-30.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-c9082--before-the-cursor-position-retry1/macos/autoedits-inline-decoration-insertion-30-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-insertion-30.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 571 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 607 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:194:5
auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines: vscode/test/e2e/auto-edits.test.ts#L154
4) auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines Error: Screenshot comparison failed: 582 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 546 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 582 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:208:5
auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines: vscode/test/e2e/auto-edits.test.ts#L154
4) auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines Retry #1 ─────────────────────────────────────────────────────────────────────────────────────── Error: Screenshot comparison failed: 582 pixels (ratio 0.01 of all image pixels) are different. Expected: /Users/runner/work/cody/cody/vscode/test/e2e/__snapshots__/auto-edits.test.ts/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png Received: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-actual.png Diff: /Users/runner/work/cody/cody/vscode/test-results/auto-edits-autoedits-trigg-d9853-required-on-different-lines-retry1/macos/autoedits-inline-decoration-multiple-insertions-different-lines-44-diff.png Call log: - expect.toHaveScreenshot(macos/autoedits-inline-decoration-multiple-insertions-different-lines-44.png) with timeout 5000ms - verifying given screenshot expectation - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - 546 pixels (ratio 0.01 of all image pixels) are different. - waiting 100ms before taking screenshot - taking page screenshot - disabled all CSS animations - waiting for fonts to load... - fonts loaded - captured a stable screenshot - 582 pixels (ratio 0.01 of all image pixels) are different. 152 | await page.waitForTimeout(500) 153 | > 154 | await expect(page).toHaveScreenshot([snapshotPlatform, snapshotName], { | ^ 155 | maxDiffPixelRatio: 0.02, 156 | maxDiffPixels: 500, 157 | clip, at autoeditsTestHelper (/Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:154:28) at /Users/runner/work/cody/cody/vscode/test/e2e/auto-edits.test.ts:208:5
fast_tests_matrix_prep
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
slow_tests_matrix_prep
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
test-integration (ubuntu)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
build
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
test-unit (ubuntu, 20)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
test-unit (ubuntu, 18)
ubuntu-latest pipelines will use ubuntu-24.04 soon. For more details, see https://github.com/actions/runner-images/issues/10636
Slow Test: vscode/chat-atFile.test.ts#L1
vscode/chat-atFile.test.ts took 1.6m
Slow Test: vscode/initial-context.test.ts#L1
vscode/initial-context.test.ts took 1.4m
Slow Test: vscode/chat-input.test.ts#L1
vscode/chat-input.test.ts took 1.2m
Slow Test: vscode/command-custom.test.ts#L1
vscode/command-custom.test.ts took 1.2m
Slow Test: vscode/command-edit.test.ts#L1
vscode/command-edit.test.ts took 53.1s
🎭 Playwright Run Summary
7 failed auto-edits.test.ts:161:1 › autoedits: triggers a multi-line diff view when edit affects existing lines auto-edits.test.ts:175:1 › autoedits: triggers an inline completion when edit is an insertion immediately after the cursor auto-edits.test.ts:189:1 › autoedits: triggers an inline decoration when an inline completion is desired, but the insertion position is before the cursor position auto-edits.test.ts:203:1 › autoedits: triggers inline decorations when multiple insertions are required on different lines auto-edits.test.ts:217:1 › autoedits: triggers inline decorations when multiple separate insertions are required on the same line auto-edits.test.ts:239:1 › autoedits: does not show any suggestion if the suffix decoration spans further than the end of the file chat-messages.test.ts:17:5 › chat assistant response code buttons ────────────────────────────── 4 skipped 46 passed (20.9m)

Artifacts

Produced during runtime
Name Size
playwright-recordings macos
13 MB