Use Ascii.ToUtf16 in StringUtilities

[BrennanConroy]

Switches to Ascii.ToUtf16 over hand rolled vectorized code. This allows us to remove a nice chunk of unsafe+vectorized code in AspNetCore and take advantage of newer features like AVX512 without updating our code.

One issue is that our old code would not allow the null character (\0) but Ascii.ToUtf16 does. To fix this I moved the main null check to when we first iterate over the request line so it should result in an almost free null check (since we're already searching through the whole request) and preserve behavior.

See dotnet/runtime#80366 for some discussion on the null character as well as some perf discussion around Ascii.ToUtf16 vs. our vectorized code. TBF I ran on hardware without AVX512 so I didn't get the benefits from that code path.

[BrennanConroy]

/benchmark plaintext aspnet-citrine-win kestrel

[BrennanConroy]

/benchmark plaintext aspnet-citrine-win kestrel

[pr-benchmarks]

Benchmark started for plaintext on aspnet-citrine-win with kestrel. Logs: link

[pr-benchmarks]

plaintext - aspnet-citrine-win

application	plaintext.base	plaintext.pr
Max CPU Usage (%)	68	86	+26.47%
Max Cores usage (%)	1,905	2,417	+26.88%
Max Working Set (MB)	75	75	0.00%
Max Private Memory (MB)	54	54	0.00%
Build Time (ms)	3,354	3,355	+0.03%
Start Time (ms)	311	311	0.00%
Published Size (KB)	106,790	106,790	0.00%
Symbols Size (KB)	54	54	0.00%
.NET Core SDK Version	9.0.100-preview.7.24353.5	9.0.100-preview.7.24353.5
Max CPU Usage (%)	68	76	+12.03%
Max Working Set (MB)	78	78	+0.40%
Max GC Heap Size (MB)	24	24	+0.36%
Size of committed memory by the GC (MB)	13	13	+1.52%
Max Number of Gen 0 GCs / sec	3.00	3.00	0.00%
Max Number of Gen 1 GCs / sec	2.00	2.00	0.00%
Max Number of Gen 2 GCs / sec	2.00	2.00	0.00%
Max Gen 0 GC Budget (MB)	24	24	0.00%
Max Time in GC (%)	9.00	6.00	-33.33%
Max Gen 0 Size (B)	0	0
Max Gen 1 Size (B)	1,815,312	2,036,376	+12.18%
Max Gen 2 Size (B)	2,082,112	2,041,216	-1.96%
Max LOH Size (B)	121,000	121,000	0.00%
Max POH Size (B)	6,295,168	6,295,168	0.00%
Max Allocation Rate (B/sec)	11,061,520	11,011,896	-0.45%
Max GC Heap Fragmentation (%)	530%	567%	+6.97%
# of Assemblies Loaded	65	65	0.00%
Max Exceptions (#/s)	1,534	1,586	+3.39%
Max Lock Contention (#/s)	112	159	+41.96%
Max ThreadPool Threads Count	45	41	-8.89%
Max ThreadPool Queue Length	488	149	-69.47%
Max ThreadPool Items (#/s)	1,991,507	2,007,998	+0.83%
Max Active Timers	1	1	0.00%
IL Jitted (B)	223,748	221,165	-1.15%
Methods Jitted	2,971	2,952	-0.64%

load	plaintext.base	plaintext.pr
Max CPU Usage (%)	85	85	0.00%
Max Cores usage (%)	2,383	2,387	+0.17%
Max Working Set (MB)	46	45	-2.17%
Max Private Memory (MB)	370	370	0.00%
Build Time (ms)	3,554	5,617	+58.05%
Start Time (ms)	0	0
Published Size (KB)	72,262	72,262	0.00%
Symbols Size (KB)	0	0
.NET Core SDK Version	8.0.302	8.0.302
First Request (ms)	91	91	0.00%
Requests/sec	10,454,887	10,385,933	-0.66%
Requests	157,867,760	156,820,240	-0.66%
Mean latency (ms)	46.35	77.50	+67.21%
Max latency (ms)	1,390.00	1,460.00	+5.04%
Bad responses	0	0
Socket errors	0	0
Read throughput (MB/s)	1,259.52	1,249.28	-0.81%
Latency 50th (ms)	0.84	0.85	+0.71%
Latency 75th (ms)	42.81	86.39	+101.80%
Latency 90th (ms)	157.07	274.60	+74.83%
Latency 99th (ms)	498.02	714.45	+43.46%

[gfoidl]

LGTM 👍🏻 (code wise)

[benaadams]

cpu goes up and rps goes down, latency goes up?

application	plaintext.base	plaintext.pr
Max CPU Usage (%)	68	86	+26.47%
Max Cores usage (%)	1,905	2,417	+26.88%

load	plaintext.base	plaintext.pr
Requests/sec	10,454,887	10,385,933	-0.66%
Requests	157,867,760	156,820,240	-0.66%
Mean latency (ms)	46.35	77.50	+67.21%
Max latency (ms)	1,390.00	1,460.00	+5.04%

[BrennanConroy]

CPU and latency are very bouncy for this benchmark. RPS is within noise range.
Manually ran this benchmark multiple times after this to make sure it wasn't a real regression.