Play wright tests are failing when running test cases through ado pipeline. Error [EPERM: operation not permitted, rename 'C:<some-path>\xx.map' -> 'C:\xy.map'

[KausikPonnapalli7]

System info

Playwright Version: [1.34.3]
Operating System: [Windows]
Browser: [Chromium]
Other info:

###Issue
When I am running the play wright test through an ado pipeline all test cases inside the test suite are failing. This is happening only for the first time pipeline run trigger. When I'm re triggering the same pipeline run this issue is not showing up.

###Error
Starting playwright: yarn playwright test x.e2e.test.ts --project=Chromium --retries=2

$ D:\data\CHT\134a2915\src\ooui\apps\package\node_modules.bin\playwright test x.e2e.test.ts --project=Chromium --retries=2

Error: EPERM: operation not permitted, rename 'C:\xx.map' -> 'C:\xy.map'

at Object.renameSync (node:fs:980:3)
at writeFileSyncAtomic (D:\data\CHT\134a2915\src\ooui.store\[email protected]\node_modules\playwright-core\lib\utils\fileUtils.js:72:15)
at addToCache (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\transform\compilationCache.js:99:47)
at transformHook (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\transform\transform.js:159:13)
at _utilsBundle.pirates.addHook.exts (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\transform\transform.js:196:12)
at Module.f._compile (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:936)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.i..ut._extensions. (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (D:\data\CHT\134a2915\src\ooui\packages\e2e-tests-utils\lib\src\login\index.ts:2:1)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module.f._compile (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:994)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.i..ut._extensions. (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (D:\data\CHT\134a2915\src\ooui\packages\e2e-tests-utils\src\index.ts:2:1)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module.f._compile (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:994)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.i..ut._extensions. (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at Object. (D:\data\CHT\134a2915\src\ooui\apps\package\playwright.config.ts:4:1)
at Module._compile (node:internal/modules/cjs/loader:1101:14)
at Module.f._compile (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:994)
at Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
at Object.i..ut._extensions. (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\utilsBundleImpl.js:16:1010)
at Module.load (node:internal/modules/cjs/loader:981:32)
at Function.Module._load (node:internal/modules/cjs/loader:822:12)
at Module.require (node:internal/modules/cjs/loader:1005:19)
at require (node:internal/modules/cjs/helpers:102:18)
at requireOrImport (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\transform\transform.js:172:20)
at requireOrImportDefaultObject (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\common\configLoader.js:121:53)
at ConfigLoader.loadConfigFile (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\common\configLoader.js:94:26)
at runTests (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\cli.js:118:55)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at i. (D:\data\CHT\134a2915\src\ooui.store@[email protected]\node_modules@playwright\test\lib\cli.js:40:7) {
errno: -4048,
syscall: 'rename',
code: 'EPERM',
path: 'C:\xx.map-beqweee',
dest: 'C:\xy.map'
}

[KausikPonnapalli7]

Previous related issue:
#24569

[aslushnikov]

Hey @KausikPonnapalli7,

I see you put "Ubuntu Server" as an operating system, but all the paths in the stack trace are windows-like. So are you running on Windows or Ubuntu?

[KausikPonnapalli7]

Hi @aslushnikov ,

We are running tests on Windows image. sorry for the confusion.

[aslushnikov]

Context: we've landed a patch to fix disk cache races for compilation cache - #26353

While it does fix the race on Linux & Mac, it fails to work properly on Windows since file rename is not reliable on Win. For example, the following snippet fails reliably on @mxschmitt's machine after 20-40 iterations:

import path from 'path';
import fs from 'fs';

function writeFileSyncAtomic(aPath: string, data: Buffer | string, options: WriteFileOptions) {
  const dirName = path.dirname(aPath);
  const fileName = path.basename(aPath);
  const tmpPath = path.join(dirName, fileName + '-' + createGuid());
  fs.writeFileSync(tmpPath, data, options);
  fs.renameSync(tmpPath, aPath);
}

for (let i = 0; i < 1000; ++i) {
  console.log(`iteration #${i}`);
  writeFileSyncAtomic('./temp.txt', 'hello', { encoding: 'utf-8'});
}

Investigation notes:

• We're not sure what causes the EPERM error. Adding exception to the Windows Defender / disabling windows indexer didn't help: the snippet still fails.
• The snippet fails even if using the write-file-atomic package
• It also turns out that file renaming on Windows is a notorious issue:
  • EPERM when renaming files on Windows nodejs/node#29481
  • win,fs: retry if uv_fs_rename fails libuv/libuv#1981
• VSCode (and looks like everybody around) landed re-tries to fix rename on Windows: fs - retry rename on Windows and adopt vscode#188899