perf: use simdutf8 to validate UTF-8 when reading files

[shulaoda]

closes #5191

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “merge” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[codspeed-hq]

CodSpeed Performance Report

Merging #5196 will not alter performance

_{Comparing shulaoda:perf/simdutf8 (b9a1d73) with main (aaaf26c)}

Summary

✅ 29 untouched benchmarks

[Boshen]

No ... we don't need a fs crate yet.

[shulaoda]

No ... we don't need a fs crate yet.

I understand, I will continue with this work tomorrow.

[overlookmotel]

Aside from the missing aarch64_neon feature, looks good.

There's little change on benchmarks, but that's expected - they don't read from disk within the measured bench loop.

@Boshen how do we go about measuring if it makes a real-world difference?

[Boshen]

missing aarch64_neon feature

[shulaoda]

missing aarch64_neon feature

already added

[Boshen]

Aside from the missing aarch64_neon feature, looks good.

There's little change on benchmarks, but that's expected - they don't read from disk within the measured bench loop.

@Boshen how do we go about measuring if it makes a real-world difference?

I tested this locally, Mac M2 and M3 are both too powerful to show any differences.