Aligned reading strategy in StreamReaderWriter causes event latency for EventPipe

[josalem]

perfview/src/FastSerialization/StreamReaderWriter.cs

Lines 791 to 814 in 8a34d2d

	// PERF policy
	// In the streaming (non-seekable) case we don't want to buffer any more data than was
	// requested and needed for alignment because this might cause the thread to block waiting
	// for the unneeded data to arrive. There is probably a better way to do this that can
	// oportunistically buffer if the data is available but this code isn't that sophisticated
	// yet.
	//
	// In the non-streaming (seekable) case we do want to buffer because that lets the
	// reader achieve higher throughput.
	int fillSize = inputStream.CanSeek ? bytes.Length : (position + minimum + (align-1)) & ~(align-1);
	for (; endPosition < fillSize; )
	{
	System.Threading.Thread.Sleep(0); // allow for Thread.Interrupt
	int count = inputStream.Read(bytes, endPosition, fillSize - endPosition);
	inputStreamBytesRead += count;
	if (count == 0)
	{
	break;
	}
	endPosition += count;
	}

The above code in StreamReaderWriter can cause latency for EventPipe. If you have infrequent events, e.g., EventCounters, and an event block ends up a length not divisible by 8, then the reader will attempt a blocking read that will not return until more data is sent. In the case of infrequent data, this can create artificial latency.

For example, let's say there's an EventCounter session that is turned on with a frequency of 5 seconds. If any of the counter events in that stream end up in an event block that is not divisible by 8, then the reader will read the entire event block and attempt to read the remaining bytes to get to a mod(8) value. This causes EventPipeEventSource to not realize the event until the next counter shows up. This artificially adds 5 seconds of latency to potentially the entire stream depending on how frequently event blocks end up indivisible by 8.

I found this issue when triaging dotnet/diagnostics#2044. A bandaid fix is to turn on CPU sampling which will send a constant data stream guaranteeing that there will be more data to meet the reader's aligned reading requirements.