WinHttpHandler callbacks to access arbitrary native WinHTTP options

[antonfirsov]

Background and Motivation

As described in #39210, we have several user scenarios which can be addressed by utilizing new native WinHTTP options. The Windows team iterates relatively fast on them, it's not always realistic and reasonable to introduce managed API-s for every new option.

Instead, I'm proposing an API to access the native Session and Request HINTERNET handles, enabling the execution of arbitrary WinHttpSetOption and WinHttpQueryOption P/Invokes for advanced users.

Challanges

• The lifecycle of the Session Handle is bound to the lifecycle of WinHttpHandler, however it is being instantiated in a lazy way
• The lifecycle of the Request Handle is bound to a single SendAsync call. (It gets created and destroyed transparenty, within the call)
• Different request Handle options need to be set/queried in different stages of the SendAsync call. For example:
  • Proxy options have to be set before sending the request
  • Querying WINHTTP_OPTION_CONNECTION_STATS_V0 only makes sense after the request have been sent.
  • Querying WINHTTP_OPTION_REQUEST_STATS allows observing some characteristics of the response. This could be only done after the response has been received.

Proposed API

I'm proposing a callback-based model to address the lifecycle challenges. Hooking into specific events of the handle lifecycle allows a flexible way to observe/change the behavior of WinHttpHandler.

public class SafeWinHttpHandle : Microsoft.Win32.SafeHandles.SafeHandleZeroOrMinusOneIsInvalid
{
    public SafeWinHttpHandle() { }

    
    // OPTIONAL: Convinience wrappers for `WinHttpSetOption` and `WinHttpQueryOption`.
    // Disadvantage: using them still requires non-trivial marshalling with MemoryMarshal, the code may be uglier than with native P/Invoke-s
    // public uint GetWinHttpOption(uint option);
    // public void SetWinHttpOption(uint option, uint optionData);
    // public void GetWinHttpOption(uint option, System.Span<byte> optionData);
    // public void SetWinHttpOption(uint option, System.ReadOnlySpan<byte> optionData);
}

public class WinHttpHandler
{
    // ...

    public Action<SafeWinHttpHandle> SessionCreatedCallback { get; set;}
    public Action<SafeWinHttpHandle, HttpRequestMessage> RequestCreatedCallback { get; set;}
    public Action<SafeWinHttpHandle, HttpRequestMessage> RequestSentCallback { get; set;}
    public Action<SafeWinHttpHandle, HttpRequestMessage> ResponseReceivedCallback { get; set;}

    // ...
}

Usage Examples

The following code observes the time since the the underlying TCP connection has been created by querying WINHTTP_OPTION_CONNECTION_STATS_V0, forcing the creation of a new TCP connection after some time elapsed. (This is very similar, but probably not identical to an actual use-case AFAIK.)

// https://docs.microsoft.com/en-us/windows/win32/api/mstcpip/ns-mstcpip-tcp_info_v0
struct TCP_INFO_v0
{
    // ...
    public UInt64 ConnectionTimeMs;
    // ...
};

[DllImport("winhttp.dll", SetLastError = true)]
internal static extern bool WinHttpSetOption(SafeWinHttpHandle handle, uint option, IntPtr optionData, uint optionLength);

[DllImport("winhttp.dll", SetLastError = true)]
internal static extern bool WinHttpQueryOption(SafeWinHttpHandle handle, uint option, IntPtr buffer, ref uint bufferSize);

const uint WINHTTP_OPTION_CONNECTION_STATS_V0 = 141;
const uint WINHTTP_OPTION_EXPIRE_CONNECTION = 143;

var handler = new WinHttpHandler
{
    RequestSentCallback = (h, r) =>
    {
        TCP_INFO_v0 info;

        unsafe
        {
            TCP_INFO_v0* ptr = &info;
            uint size = (uint)sizeof(TCP_INFO_v0);
            if (!WinHttpQueryOption(h, WINHTTP_OPTION_CONNECTION_STATS_V0, (IntPtr)ptr, ref size))
            {
                throw new Exception("WinHttpQueryOption failed! " + Marshal.GetLastWin32Error());
            }
        }

        if (info.ConnectionTimeMs > 60000)
        {
            if (!WinHttpSetOption(h, WINHTTP_OPTION_EXPIRE_CONNECTION, IntPtr.Zero, 0))
            {
                throw new Exception("WinHttpSetOption failed! " + Marshal.GetLastWin32Error());
            }
        }
    }
};

Alternative Designs

• Expsoing protected virtual methods instad of callbacks
• Session Handle can be probably exposed as a property, but it seemed more consistent, and less risky to integrate it into the event-based model.

Risks

It doesn't seem trivial to me whethere there is a "right spot" to invoke the RequestSentCallback and ResponseReceivedCallback to enable querying/setting any current and future option. A mistake here can lead to breaking changes or the need for further callbacks.

runtime/src/libraries/System.Net.Http.WinHttpHandler/src/System/Net/Http/WinHttpHandler.cs

Lines 870 to 911 in 0d62c36

	using (state.CancellationToken.Register(s => ((WinHttpRequestState)s).RequestHandle.Dispose(), state))
	{
	do
	{
	_authHelper.PreAuthenticateRequest(state, proxyAuthScheme);
	await InternalSendRequestAsync(state);
	if (state.RequestMessage.Content != null)
	{
	await InternalSendRequestBodyAsync(state, chunkedModeForSend).ConfigureAwait(false);
	}
	bool receivedResponse = await InternalReceiveResponseHeadersAsync(state) != 0;
	if (receivedResponse)
	{
	// If we're manually handling cookies, we need to add them to the container after
	// each response has been received.
	if (state.Handler.CookieUsePolicy == CookieUsePolicy.UseSpecifiedCookieContainer)
	{
	WinHttpCookieContainerAdapter.AddResponseCookiesToContainer(state);
	}
	_authHelper.CheckResponseForAuthentication(
	state,
	ref proxyAuthScheme,
	ref serverAuthScheme);
	}
	} while (state.RetryRequest);
	}
	state.CancellationToken.ThrowIfCancellationRequested();
	// Since the headers have been read, set the "receive" timeout to be based on each read
	// call of the response body data. WINHTTP_OPTION_RECEIVE_TIMEOUT sets a timeout on each
	// lower layer winsock read.
	uint optionData = unchecked((uint)_receiveDataTimeout.TotalMilliseconds);
	SetWinHttpOption(state.RequestHandle, Interop.WinHttp.WINHTTP_OPTION_RECEIVE_TIMEOUT, ref optionData);
	HttpResponseMessage responseMessage =
	WinHttpResponseParser.CreateResponseMessage(state, _doManualDecompressionCheck ? _automaticDecompression : DecompressionMethods.None);
	state.Tcs.TrySetResult(responseMessage);

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[antonfirsov]

I'm closing this because of the reasons described in #39210 (comment).
Unless we see stronger customer need for a generic solution, we will keep exposing individual options for now like #44025