Why fs.promises.writeFile write buffer multi times with 512KiB limit for each time?

[LongTengDao]

Why not write the whole buffer one time?

Won't that be faster in fact?

[Ayase-252]

Could you provide which code snippet in question?

[aduh95]

Why not write the whole buffer one time?

I think Node.js tries to write the whole buffer, but depending on the host OS/FS/hardware, it might need to write in several bits. How did you come with this 8192 figure?

Won't that be faster in fact?

It certainly would be, but I'm afraid there's nothing we can do at Node.js level.

[RaisinTen]

The only place where I could find 8192 related to fs is in the implementation of readFileSync:

node/lib/fs.js

Lines 427 to 428 in a0261d2

	buffer = Buffer.allocUnsafe(8192);
	bytesRead = tryReadSync(fd, isUserFd, buffer, 0, 8192);

If the file being read from is not a regular file (we can't tell the file size from statSync if it is not regular, stdin for instance), we create a buffer of size 8192 (value of the macro BUFSIZ of glibc) and read into it. We continue reading and appending the read buffer to an array of buffers while the number of bytes read (returned by tryReadSync) is not 0.

[LongTengDao]

Could you provide which code snippet in question?

I think Node.js tries to write the whole buffer, but depending on the host OS/FS/hardware, it might need to write in several bits. How did you come with this 8192 figure?
It certainly would be, but I'm afraid there's nothing we can do at Node.js level.

The only place where I could find 8192 related to fs is in the implementation of readFileSync:

node/lib/fs.js

Lines 427 to 428 in a0261d2

	buffer = Buffer.allocUnsafe(8192);
	bytesRead = tryReadSync(fd, isUserFd, buffer, 0, 8192);

node/lib/internal/fs/promises.js

Line 298 in a0261d2

MathMin(kWriteFileMaxChunkSize, data.byteLength));

Sorry, I made a mistake while reading the source, in fact only fs.promises.writeFile (which I usually use) has a voluntary limit. And the limit is 512KiB, not 8192:

node/lib/internal/fs/utils.js

Line 136 in a0261d2

const kWriteFileMaxChunkSize = 512 * 1024;

[Linkgoron]

The writeFile in the promises version does indeed write in chunks, this is in contrast to the callback version which doesn't. I'm not sure why this was done, but I assume that it was done in order to imitate what's done on readFile (#25741 and https://nodejs.org/api/fs.html#fs_performance_considerations).

Note that prior to Node 15.12 the limit was 16 kbytes, and now it's 512 kbytes - so it should be more performant in newer versions.