How to create HttpClientHandler later than configuration time

[Timovzl]

As dicussed briefly in #71:

Looking at the usage of services.AddHttpClient().ConfigurePrimaryHttpMessageHandler(), I'm wondering how to configure it when the necessary information is not available at configuration time.

Say that each time we want to use the HttpClient, we load a certificate (perhaps based on a fingerprint), with the intention of getting an HttpClientHandler that uses that particular certificate. Such information may not be available (or even remotely convenient) at configuration time, let alone for each endpoint.

How can we get a properly managed HttpClientHandler at a later stage, when we have all the information (including potential certificates) about a particular endpoint we are connecting to?

Note that we want the client handler's lifetime to be handled for us: don't create a new one for every request, to avoid opening tons of TCP connections, but don't keep one around indefinitely either, because DNS changes must be accounted for.

[roc916]

Seems to have relevance #932.

[davidhagg]

We also have this issue. We act as a provider for our partners and we wrap calls to a 3rd party service that requires us to have a unique client certificate for each partner we are sending the request on behalf of.

The problem is that the client certificate is set in the ConfigurePrimaryHttpMessageHandler() in the startup sequence. We store the client certificates in Azure Keyvault and will resolve these at runtime depending on the partner.

We could of course register a named client for each partner we have (about 180 today) and resolve the httpclient by name. This however requires us to restart the service each time a new partner is onboarded.

What are your recommendations on how to design this?

[davidhagg]

Any thoughts on this @Eilon?

[Eilon]

Sorry @davidhagg I'm not an expert in this area. @rynowak - can you take a look?

[aarro]

We also have this issue. We act as a provider for our partners and we wrap calls to a 3rd party service that requires us to have a unique client certificate for each partner we are sending the request on behalf of.

The problem is that the client certificate is set in the ConfigurePrimaryHttpMessageHandler() in the startup sequence. We store the client certificates in Azure Keyvault and will resolve these at runtime depending on the partner.

We could of course register a named client for each partner we have (about 180 today) and resolve the httpclient by name. This however requires us to restart the service each time a new partner is onboarded.

What are your recommendations on how to design this?

I have the exact same situation...I need to load a cert at runtime from a secret vault based on context. The solution of using the service provider inside ConfigurePrimaryHttpmessageHandler()(#932 (comment)) is sort of doable in that it provides a cleaner way of getting a cert, but there's no async capability and it makes context hacky.

I would also love some recommendations around this.

[davidhagg]

@davidfowl any suggestions/ideas?

[davidhagg]

@rynowak it would be very helpful to get your insights on this issue. If there is a work around or if we simply must live with it because it is by design. I appreciate any answer :-)

[rynowak]

We don't have a feature that does this yet sorry.

I'm interested in your ideas about how this would behave, be implemented, and how you'd want to use it. What I think you're asking for the is the ability to pass additional information besides the name to a call to CreateClient and have it do something different - in this case use a different certificate.

The problem that this creates is that the name is the key. This is fundamental unit of both configuration and caching for HttpClient factory. So if you want to use a single name and multiple client-certificates, then the HttpClient factory system has to be aware of how that works - basically we would need to do have a compound key.

I think the reason why we're arriving here is that you want to create a separate HttpClientHandler instance for each remote host/cert pair - and you want the HttpClient factory to be aware of that requirement.

I want to try and nail down the requirements of this in more detail and also loop in some HttpClient experts.

@karelz @stephentoub @davidsh

[davidhagg]

Thanks for the update @rynowak. I understand that the current design makes it hard to change this since the pooling of handlers is one of the good things that were introduced with the new HttpClientFactory.

I'll try to explain our problem.

We communicate with a 3rd party service where we basically start a "session" on behalf of our customers. Each of our customers give us the client certificate that must be provided in order to communicate with the 3rd party service. The communication with the 3rd party service is done through a class in our solution that takes a HttpClient injected into the constructor. We keep all our customers client certificates in an Azure keyvault and the intention is to read the correct certificate depending on the incoming request data (i.e. which customers cert we should be using)

So the issue we are facing is that the factory to create a HttpClientHandler is setup at startup and this is where we set the client certificate to use.

Since we are using the same class to call our 3rd party service for many different customers we can't use the option to use the standard HttpClientFactory with a delegate to create a messagehandler like:
services.AddHttpClient().ConfigurePrimaryHttpMessageHandler(() => { ... })
since it would simply be impossible to distinguish the handler to use in the pool of handlers.

This pretty much leaves us with the option to have a named client for every customer. This would result in about 180 named clients (and hopefully more over time). Each with their own pool of handlers connected to a client certificate for that customer. This might be feasible but the big issue with this solution is that we will have to restart our application each time we onboard a new customer since that would require this code to be run again:

services.AddHttpClient("CustomerName").ConfigurePrimaryHttpMessageHandler(sp =>
{
    var certificateProvider = sp.GetService<CertificateProvider>();

    var handler = new HttpClientHandler();
    handler.ClientCertificates.Add(certificateProvider.GetCertificate("CustomerName"));
    return handler;
});

[aarro]

I think the reason why we're arriving here is that you want to create a separate HttpClientHandler instance for each remote host/cert pair - and you want the HttpClient factory to be aware of that requirement.

Honestly I'd rather not have to deal with that. Speaking purely from the perspective of a developer wanting to make HTTP calls on an API, ignoring any preconceived notions or valid architecture issues that make this a bad idea, it sure is nice the way curl does it.

 curl -X GET \
   {url} \
   --key pk.pem \
   --cert cert.pem \
   --cacert ca.pem \ 
   -H 'Content-Type: application/json'

In curl if I need to include a private key or specific cert(s), I just add a flag.

In dotnet, I would see this as similar to Headers or Content on an HttpRequestMessage...a property or series of properties to provide certs for a single request.

This would require some framework logic (in the handler I assume) to recognize that I'm providing explicit certs and use them. But, it would allow the developer to implement the logic for finding and assigning the cert(s) inside a Typed Client, the ideal place to do this in my opinion. Once loaded from wherever the data is stored, it's as simple as adding a header or replacing a portion of the URL to include the certs.

I was perhaps misguided and am admittedly not a certificate expert, but this is how I assumed it would work when I started digging in, that I'd find and load certs from my Typed Client. Obviously curl and dotnet are not equal, but being able to execute a semi-basic curl and have my request work vs what it takes to make the same request happen in dotnet core is frustrating.

Thanks for the help @rynowak (and everyone else) on this.

[Timovzl]

I'm interested in your ideas about how this would behave, be implemented, and how you'd want to use it.

Having similar (albeit milder) requirements as @davidhagg, I am using a custom implementation in production. I will post it as inspiration for a suitable solution.

Before I go into details, I will argue that we are wrestling with the design of HttpClient and HttpClientHandler here. Ideally, we would either (1) create and dispose an HttpClient instance per call, or (2) we would inject a single, reusable instance that does not contain any endpoint-specific state. In either case, the pooling of the underlying TCP connection and whatnot would be hidden from us. ADO.NET basically uses option 1 described here (with the connection string as the unique identifier), and that has become conventional. That is want we want.

Anyway, things being as they are, here is the solution I am using to both prevent running out of sockets and still have configurability at any time:

	public interface IHttpClientFactory
	{
		HttpClient CreateClient();
		HttpClient CreateClient(HttpClientPurpose purpose);
	}
	// Factory implementation omitted for brevity, but described below
	public class HttpClientPurpose
	{
		public override bool Equals(object other) => other is HttpClientPurpose typedOther && String.Equals(this.UniqueName, typedOther.UniqueName, StringComparison.OrdinalIgnoreCase);
		public override int GetHashCode() => StringComparer.OrdinalIgnoreCase.GetHashCode(this.UniqueName);

		public string UniqueName { get; }
		public Func<HttpMessageHandler> MessageHandlerFactory { get; }

		public HttpClientPurpose(string uniqueName, Func<HttpMessageHandler> messageHandlerFactory)
		{
			this.UniqueName = uniqueName ?? throw new ArgumentNullException(nameof(uniqueName));
			this.MessageHandlerFactory = messageHandlerFactory ?? throw new ArgumentNullException(nameof(messageHandlerFactory));
		}

		internal static HttpClientPurpose GenericPurpose { get; } = new HttpClientPurpose("InternalGenericPurpose", 
#if NETSTANDARD
			() => new HttpClientHandler()
#else
			() => new SocketsHttpHandler()
			{
				AutomaticDecompression = DecompressionMethods.GZip,
			}
#endif
		);
	}

• The factory implementation will be temporarily caching message handlers per "purpose" (usually an endpoint) as described below.
• The factory can create a client for a specific purpose, or a general-purpose one (parameterless overload).
• The general-purpose overload simply passes HttpClientPurpose.GenericPurpose to the other overload. It is suitable for all requests that need no customization on the message handler level.
• The HttpClientPurpose takes a factory Func to create the underlying HttpClientHandler (or actually its base type, HttpMessageHandler), which can specify a connection timeout, a client certificate, etc.
• The HttpClientPurpose takes a unique name, letting the factory know which purposes are the same and can thus use the same cached message handler.
• The HttpClientPurpose can be constructed at any point. In other words, we can load certificates at will and construct a purpose that utilizes them.
• The factory implementation stores message handlers in an IMemoryCache for 240 seconds (matching the default time TCP connections are kept open). On eviction from the cache, it moves them to a second cache, which holds "expiring" message handlers. On eviction from there after another 240 seconds, it disposes them. The purpose of the second cache is to delay the disposal of handlers that may still be working on a request that was started when it was in the first ("active") cache.
• Clearly, the disadvantage here is that message handlers could technically be disposed while the client is still using them, i.e. if a request takes more than 240 seconds (I hope you're not doing that) or if the client is keeping HttpClients around.
• The second cache should probably be replaced by another system, as items could be evicted prematurely (i.e. while still in use) if memory is running out.

[tsaluszewski]

We are having a similar issue, we are utilizing the IHttpClientFactory to avoid opening unnecessary TCP connections (we had this issue before with per request created client).

Our requirement is to use Digest authentication when the actual credentials should be set at run-time at each request rather at configure time. See below:

• DI services configuration

public static ServiceCollection CreateServices()
 {
         var serviceCollection = new ServiceCollection();
         serviceCollection.AddHttpClient("WithDigestAuthentication")
                .ConfigurePrimaryHttpMessageHandler(() => new DigestAuthenticationHandler());
         return serviceCollection;
 }

• Custom message handler

public class DigestAuthenticationHandler : HttpClientHandler
    {
        public const string Username = "Digest.Username";
        public const string Password = "Digest.Password";

        private const string AuthType = "Digest";

        protected override async Task<HttpResponseMessage> SendAsync(
            HttpRequestMessage request,
            CancellationToken cancellationToken)
        {
            // Nasty hack to get around
            // "Properties can only be modified before sending the first request." error :(
            if (this.Credentials == null)
            {
                var credCache = new CredentialCache();
                var networkCredential = new NetworkCredential(
                    request.Properties[Username].ToString(),
                    request.Properties[Password].ToString());

                credCache.Add(request.RequestUri, AuthType, networkCredential);
                this.Credentials = credCache;
            }

            return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
        }
    }

• Code that adds actual credentials to the HttpRequestMessage.Properties as advised in Make HTTP requests using IHttpClientFactory in ASP.NET Core

        private HttpRequestMessage WithCredentials(HttpRequestMessage request)
        {
            request.Properties[DigestAuthenticationHandler.Username] = this.username;
            request.Properties[DigestAuthenticationHandler.Password] = this.password;
            return request;
        }

The problem is that once HttpMessageHandler.Credentials are set these cannot be re-set on a subsequent request.

Can you please advise how to overcome this issue?
We thought about creating a pool of preconfigured "WithDigestAuthenticationXYZ" HttpClients and selecting one at runtime via our custom client factory method that would check whether given client Credentials are equal. When the pool is drained out we can revert to just returning a new instance of HttpClient. What do you think?

[Timovzl]

@tsaluszewski Beware that if you you use pooling, you probably want new clients every now and then, because otherwise DNS changes will not be taken into account. Also, I'm not sure if long-lived could break, for example if a firewall ends a TCP connection after 15 idle minutes. (But perhaps they know how to deal with that.)

I am quite satisfied with the results of the solution I posted just above your comment. It has been in used for months. I recommend a careful read of that post. It might just help complete your implementation.

[tsaluszewski]

Thank you @Timovzl for you input! However I do not want to new clients every now and then, as I had it before and ran into too many open handles (over 13K) in linux container in AWS ECS :(
Below is the solution that works for me, and maybe someone can re-use it:

public class DigestAuthenticationHandler : HttpClientHandler
    {
        public const string Username = "Digest.Username";
        public const string Password = "Digest.Password";

        private const string AuthType = "Digest";

        private readonly ILogger logger;
        private readonly SemaphoreSlim semaphore = new SemaphoreSlim(1, 1);

        public DigestAuthenticationHandler(ILogger logger)
        {
            this.logger = logger;
        }

        protected override async Task<HttpResponseMessage> SendAsync(
            HttpRequestMessage request,
            CancellationToken cancellationToken)
        {
            if (request.Properties.TryGetValue(Username, out var userValue) && string.IsNullOrEmpty(userValue as string))
            {
                throw new HttpClientException($"{Username} is not specified!");
            }

            if (request.Properties.TryGetValue(Password, out var passwordValue) && string.IsNullOrEmpty(passwordValue as string))
            {
                throw new HttpClientException($"{Password} is not specified!");
            }

            var (username, password) = ((string)userValue, (string)passwordValue);

            await this.semaphore.WaitAsync(cancellationToken).ConfigureAwait(false);
            try
            {
                if (this.Credentials == null)
                {
                    this.Credentials = CreateCredentials(request, username, password);
                    return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
                }
                else if (HasSameCredentials(this.Credentials, request, username, password))
                {
                    return await base.SendAsync(request, cancellationToken).ConfigureAwait(false);
                }
                else
                {
                    this.logger.Error(
                        "Pooled HttpClient for {0} has different credentials than requested! "
                        + "Creating new unmanaged HttpClient!", request.RequestUri);

                    return await SendWithNewClientAsync(request, cancellationToken, username, password).ConfigureAwait(false);
                }
            }
            finally
            {
                this.semaphore.Release();
            }
        }

        private static async Task<HttpResponseMessage> SendWithNewClientAsync(
            HttpRequestMessage existingRequest,
            CancellationToken cancellationToken,
            string username,
            string password)
        {
            // New request required due to "The request message was already sent. Cannot send the same request message multiple times."
            // https://stackoverflow.com/questions/18000583/re-send-httprequestmessage-exception/18014515
            var request = await existingRequest.CloneAsync().ConfigureAwait(false);
            var handler = new HttpClientHandler { Credentials = CreateCredentials(existingRequest, username, password) };
            var client = new HttpClient(handler);

            return await client.SendAsync(request, cancellationToken).ConfigureAwait(false);
        }

        private static CredentialCache CreateCredentials(HttpRequestMessage request, string username, string password)
        {
            var credCache = new CredentialCache();
            var networkCredential = new NetworkCredential(username, password);
            credCache.Add(request.RequestUri, AuthType, networkCredential);
            return credCache;
        }

        private static bool HasSameCredentials(ICredentials credentials, HttpRequestMessage request, string username, string password)
        {
            var cred = credentials.GetCredential(request.RequestUri, AuthType);
            return cred.UserName == username && cred.Password == password;
        }
    }
}

Note that request.Properties needs to be set while creating a Digest request, like below:

        private HttpRequestMessage WithCredentials(HttpRequestMessage request, string username, string password)
        {
            request.Properties[DigestAuthenticationHandler.Username] = username;
            request.Properties[DigestAuthenticationHandler.Password] = password;
            return request;
        }

[Timovzl]

@tsaluszewski Thanks for sharing your solution as well.

However I do not want to new clients every now and then, as I had it before and ran into too many open handles (over 13K) in linux container in AWS ECS :(

My solution specifically uses a strategy to avoid that. Before, we were simply creating and disposing an HttpClient per call, and the exact same problem is what led me to creating the solution I described.

Now, each handler lives for 2 * 240 minutes. For each distinct configuration (i.e. for each endpoint in the very worst case, but most endpoints simply share the default configuration in practice), there are at most two handlers at any given time.

[rynowak]

We're going to look into solve these problems via some mechanism in 5.0.

The problem as I understand it:

• HttpClientHandler/SocketsHttpClientHandler associate authN/proxy/network related settings with the handler instance
• IHttpClientFactory is a framework for configuring the outgoing http stack
• IHttpClientFactory also encourages reuse of the underlying handlers

These three points are in conflict 😆

[LeaFrock]

• HttpClientHandler/SocketsHttpClientHandler associate authN/proxy/network related settings with the handler instance
• IHttpClientFactory is a framework for configuring the outgoing http stack
• IHttpClientFactory also encourages reuse of the underlying handlers

These three points are in conflict 😆

Would you please explain more details? For me, reusing handlers with IHttpClientFactory should be a must. The users shouldn't worry about the underlying problems such as 'Socket Exhaustion'.

[Timovzl]

I would summarize the problem as: To avoid socket exhaustion, IHttpClientFactory is a necessity. However, we tend to need different configurations (client certificate, timeout, ...) for different endpoints, and we do not always have this information available on startup.

[stephentoub]

To avoid socket exhaustion, IHttpClientFactory is a necessity

Can someone summarize for me why this is believed to be the case? Is it because a ton of distinct components all want to share the same instance and can't otherwise?

[rynowak]

To avoid socket exhaustion, IHttpClientFactory is a necessity

Can someone summarize for me why this is believed to be the case? Is it because a ton of distinct components all want to share the same instance and can't otherwise?

@stephentoub

Keeping an instance around forever wasn't a good solution in 2.0 when this feature was added because you'd end up with stale DNS. Keep in mind that we're talking about environments where you keep HTTP connections open for days.

Now that we have SocketsHttpHandler you can do this and still avoid the stale-DNS problem by configuring PooledConnectionLifetime. However, we haven't done enough to get the word out, so people are still convinced that the client factory is the only solution. The property has no documentation at all 😢 I just sent a PR to suggest this in the client factory docs dotnet/AspNetCore.Docs#15854

---

• HttpClientHandler/SocketsHttpClientHandler associate authN/proxy/network related settings with the handler instance
• IHttpClientFactory is a framework for configuring the outgoing http stack
• IHttpClientFactory also encourages reuse of the underlying handlers
  These three points are in conflict 😆

Would you please explain more details? For me, reusing handlers with IHttpClientFactory should be a must. The users shouldn't worry about the underlying problems such as 'Socket Exhaustion'.

@LeaFrock

Sure.

A client handler (SocketsClientHandler) is for all intents and purposes a connection pool. It represents a set of a connections to a set of hosts. This is why keeping a long-lived one, or reusing/sharing one is efficient, because you can share the connections you don't have to wait for them to be reestablish.

Features like client certificates or proxy settings describe how those connections are made. You cannot share the same connections to the same host with two different client certificates - you're asking to do something impossible.

Since in .NET a client handler represents a pool of connections, that is where settings related to client certificates and proxy servers life. You could argue that it's inconvenient for you, but that's the design .NET has.

So back to my three statements.

• Users want to multiplex proxies and certificates (including multiple certs and proxies to the same hosts)
• They want this to be easy to code up (using client factory)
• They want this to be efficient (reuse connections where possible)

These desires conflict with how all of this is designed.

[stephentoub]

Now that we have SocketsHttpHandler you can do this and still avoid the stale-DNS problem by configuring PooledConnectionLifetime

Right. That's why I was asking :) Thanks.

[rynowak]

Now that we have SocketsHttpHandler you can do this and still avoid the stale-DNS problem by configuring PooledConnectionLifetime

Right. That's why I was asking :) Thanks.

It would be great to see improved CoreFx docs on this, or something else to help get the word about about how users should using HttpClient in 2019 (blog)

[stephentoub]

It would be great to see improved CoreFx docs on this

I agree.

cc: @karelz

[LeaFrock]

@rynowak Thank you very much.

• Users want to multiplex proxies and certificates (including multiple certs and proxies to the same hosts)

That's the scenario we meet and that's why we're here to discuss this question.

• They want this to be easy to code up (using client factory)

Certainly we hope to code up easily, but not define that this problem must be resolved by using client factory. I thought some implementation of IHttpClientFactory could help, but now I understand I'm wrong.

• They want this to be efficient (reuse connections where possible)

That's the goal. Now we have to create a lot of HttpClient instances for each proxy which may cause 'Socket Exhaustion' or else. Also, if we want to be as efficient as possible currently, the instances have to be manually managed, maybe in a dictionary(Key-Value : Proxy/Cert-HttpClient). When the proxy/cert is not available(maybe out of date or else), we need to remember that the HttpClient instance should be removed and disposed. I think it's not elegant, and there should be some official best practice to inform the developers how to deal with this problem. We hope our codes run safely and efficiently in this scenario.

Glad to see that DNC notices the problem, and I'm looking forward to a better solution.

[Timovzl]

Now that we have SocketsHttpHandler you can do this and still avoid the stale-DNS problem by configuring PooledConnectionLifetime.

@rynowak Thank you for clearing that up! That definitely helps solve the problem. I had no idea that could serve as an alternative to the factory.

A few follow-up questions just to make sure that we use this correctly:

From the updated docs:

• Create an instance of SocketsHttpHandler when the app starts and use it for the life of the app.
• Configure xref:System.Net.Http.SocketsHttpHandler.PooledConnectionLifetime to an appropriate value based on DNS refresh times.
• Create HttpClient instances using new HttpClient(handler, dispostHandler: false) as needed.

1. We could technically create the SocketsHttpHandler at any time, such as lazily if that helps, correct? As long as we create one per uniquely configured endpoint and reuse it.
2. We only need a separate SocketsHttpHandler per configuration, correct? E.g. if we have two endpoints that each use no certificate and no other alternative configuration, we could simply use the same instance for them if we wanted to?
3. For PooledConnectionLifetime, what would generally be "an appropriate value based on DNS refresh times"?
4. What is the default value for PooledConnectionLifetime? If it does have a proper default, we can keep our hands of such details, which protects us from trying to configure things we have little knowledge about.

[stephentoub]

We could technically create the SocketsHttpHandler at any time, such as lazily if that helps, correct?

Correct. Just keep in mind that it is the handler that owns its own connection pooling resources, so a) you still want to limit the number you create to as few as you need (if you need another because you want different settings applied to the pooled connections, that's a good reason to create another), and b) you want to Dispose of the handler when you're done with it if the app is going to keep on running after, as otherwise connections might end up staying open longer than you'd like.

We only need a separate SocketsHttpHandler per configuration, correct? if we have two endpoints that each use no certificate and no other alternative configuration, we could simply use the same instance for them if we wanted to?

Correct. And in fact in the situation you cite, you really should strive to use the same instance. Otherwise, you may end up with more connections than is your ideal minimum.

For PooledConnectionLifetime, what would generally be "an appropriate value based on DNS refresh times"?

Using the pooled connection lifetime to avoid stale DNS, you'd want to factor in how common it is for your target server's DNS information to change. In general with DNS TTL, an hour is considered short and a day is considered long, though in the wild TTLs can be found to range anywhere from a minute to a few days. At the same time, connection pooling is used in order to minimize the costs of establishing new connections, but even though establishing connections is relatively expensive when compared to just using an existing one, the overhead we're generally talking about is measured anywhere from milliseconds to seconds, so even with a really short connection lifetime, you're unlikely to see the overheads show up, and you can tune your app as you see fit without feeling badly about it. In fact, if memory serves, HttpClientFactory defaults to something around 2 minutes.

What is the default value for PooledConnectionLifetime?

Infinite, as by default SocketsHttpHandler wants to avoid tearing down connections that are actively being used over and over. However, there's also PooledConnectionIdleTimeout, which is how long a connection can remain open in the pool unused before it's closed, and that defaults to 2 minutes.

[Timovzl]

@stephentoub Many thanks!

@LeaFrock I'm closing this issue, as I consider it adequately covered. If you have any unanswered follow-ups, we can always re-open it.