#2119 Review load balancing (2nd round) and redesign DefaultConsulServiceBuilder with ConsulProviderFactory refactoring to make it thread safe and friendly

[raman-m]

Fixes #2119

• Concurrent requests with different services, using consul, returns the same service and then 404 #2119

Root Cause

In commit 34cb3eb by PR #2067, new static definitions were introduced in ConsulProviderFactory due to the DefaultConsulServiceBuilder constructor needing a factory argument to obtain a ConsulRegistryConfiguration object for implementing the IConsulServiceBuilder.Configuration property. The static design of ConsulProviderFactory necessitated retrieving the service builder from the service provider to inject into the Consul object. Consequently, a Func<ConsulRegistryConfiguration> service was added to the DI to enable the acquisition of the config object in the DefaultConsulServiceBuilder constructor. However, the delegate was static, which was a significant flaw in attempting to design the feature with static definitions, leading to only the first request receiving the correct config object for the first service name. Subsequent requests with another service name failed to receive the correct config object.

To be honest, my grasp of this bug is not yet complete... Here is clean branch created from develop with tests that replicate the issue: raman-m/2119-steps-to-repro-clean-experiment.
However, the acceptance test is now passing, and I am optimistic that we can proceed with the merge.

Proposed Changes

• In the Consul class, the service collection manipulation was removed due to its lack of thread safety. It is preferable to return the resulting collection after the service building stage.
• In the ConsulProviderFactory class, I removed the static definitions in favor of propagating the configuration object through HttpContext items because I've now made the service builder a scoped service. This ensures that the configuration will be correct for future users' overridden descendants.
• In the DefaultConsulServiceBuilder class, the class has been scoped as a service, prompting the injection of the IHttpContextAccessor service. This allows access to the true-scoped HttpContext object and enables reading the correct configuration object from items.
• In the OcelotBuilderExtensions class, the IConsulServiceBuilder service was modified to be a scoped service.
• Reviewed load balancer feature
• Added ConcurrentSteps to develop concurrency-tests in acceptance testing.

Predecessors

• #954 #957 #1026 Customize Consul services creation in Consul service discovery provider #2067 in release 23.3.0
• #1634 #1487 #1329 #1304 #1294 #793 Consul polling of services: enhancements and fix errors #1670 in release 20.0.0

[raman-m]

@ggnaegi Hi Gui,
I would value your input on a code review.

I've established a clean branch from develop with tests that replicate the issue: raman-m/2119-steps-to-repro-clean-experiment
Please focus on the ShouldReturnDifferentServicesWhenConcurrentlyRequestingToDifferentServices test, which consistently fails, so it replicates the bug.
This branch will be used to dissect the concurrency bug and better articulate the root cause.
If appropriate, we could utilize this branch for additional analysis and compare it to the bugfix changes in the feature branch.
I'll conduct a thorough analysis later; however, you may commence the code review immediately.

[raman-m]

Task

• Add tests from clean experiment "raman-m/2119-steps-to-repro-clean-experiment" → ✔️ commit eb9be3e

[raman-m]

@ggnaegi Fascinating design, isn't it? 😄 We're upgrading from the In-Memory poller to this advanced one. Exciting!

[raman-m]

@ggnaegi With the addition of commit eb9be3e, I am now fully confident that the proposed hotfix is effective. 😃 Looking forward to your review... 🙏
📓 The primary acceptance test is ShouldReturnDifferentServicesWhenConcurrentlyRequestingToDifferentServices.

[raman-m]

Development Complete❗

@ggnaegi @RaynaldM Please review and approve officially.

@minnocenti901 Your assistance with testing would be valuable. Could you please create a local build from the feature branch, deploy it, and conduct basic routing tests from #2119 ?

[ggnaegi]

Ok, good idea to pass the configuration to the http context items. Are we sure it's sound in case of multiplexing too?

[ggnaegi]

Agree, it's not good at the minute.

[raman-m]

Indeed, it is necessary. However, there is no time to refactor further because it's beyond the scope of the current bug fix. For now, the comments in src are noted for future refactoring. The present solution may not be elegant, but it functions. I have prepared a TODO task for the next phase of refactoring, which has been added to #2141 (comment) (item 5).

Not an issue.

[ggnaegi]

Great idea to use the http context items as storage of the consul registry configuration. Just a stupid question. Could we have some side effects with the multiplexer (copies of the http context)?

[raman-m]

I know, you worry about this baby, your beloved multiplexer... 🤣 ❤️
Good question... The IServiceDiscoveryProviderFactory is incorporated within the LoadBalancerFactory, which is invoked by the LoadBalancerHouse and is a component of the LoadBalancingMiddleware. Consequently, all service discovery providers, including our lovely Consul provider, are part of the LoadBalancingMiddleware.
To address your question, we must analyze the differences between LoadBalancingMiddleware and MultiplexingMiddleware, as well as their respective positions in the Ocelot pipeline.

Ocelot/src/Ocelot/Middleware/OcelotPipelineExtensions.cs

Line 27 in 58d87c9

public static RequestDelegate BuildOcelotPipeline(this IApplicationBuilder app,

Ocelot/src/Ocelot/Middleware/OcelotPipelineExtensions.cs

Line 59 in 58d87c9

app.UseMultiplexingMiddleware();

Ocelot/src/Ocelot/Middleware/OcelotPipelineExtensions.cs

Line 134 in 58d87c9

app.UseLoadBalancingMiddleware();

In the method on line 190, the early bird multiplexer invokes the load-balancing middleware, as seen here:

Ocelot/src/Ocelot/Multiplexer/MultiplexingMiddleware.cs

Lines 184 to 192 in 58d87c9

	private async Task<HttpContext> ProcessRouteAsync(HttpContext sourceContext, DownstreamRoute route, List<PlaceholderNameAndValue> placeholders = null)
	{
	var newHttpContext = await CreateThreadContextAsync(sourceContext, route);
	CopyItemsToNewContext(newHttpContext, sourceContext, placeholders);
	newHttpContext.Items.UpsertDownstreamRoute(route);
	await _next.Invoke(newHttpContext);
	return newHttpContext;
	}

Subsequently, the next delegate is passed a new context object, correct? Thus, the LoadBalancingMiddleware is provided with a new HttpContext object. However, it appears inconsequential whether a new or old context object is processed due to the injection of IHttpContextAccessor?...
The questions could be phrased as: "Do the IHttpContextAccessor instances in both ConsulProviderFactory and DefaultConsulServiceBuilder reference the same HttpContext objects?'
It seems irrelevant whether they are new or old. Ideally, given the load balancing context, the same HttpContext should be maintained.

I've reviewed the DI container setup, and it appears that our IHttpContextAccessor service is indeed a singleton as shown here:

Ocelot/src/Ocelot/DependencyInjection/OcelotBuilder.cs

Line 125 in 58d87c9

Services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();

It seems we're processing an old context object, not the one from the multiplexer, which is disappointing. However, this should not be an issue since we use the same context object in both ConsulProviderFactory and DefaultConsulServiceBuilder, so there shouldn't be any exceptions or empty configuration objects during the read operation: DefaultConsulServiceBuilder Line#30

And...

Ocelot/src/Ocelot/DependencyInjection/OcelotBuilder.cs

Line 125 in 58d87c9

Services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();

Ups❗ Why is it added as singleton? 🙄 I thought it is scoped service. 🙈 "Houston, we have a problem!" 🤣

Oh, no....
https://github.com/ThreeMammals/Ocelot/blame/develop/src/Ocelot/DependencyInjection/OcelotBuilder.cs#L125

[raman-m]

Tom added this service as singleton in 2017... 🤯

• commit 6289992#diff-8f16fc8879d2a7b30200bf1a3aafbb7baf957f49c53db5db704a3c3fbd985c71R119
• PR added Ocelot bulder to try and make adding Ocelot configuration more … #159

Oh, no!... This problem exists since ages ❗ 🤕

[raman-m]

My Visual Studio disassembler shows this code:

    /// <summary>
    /// Provides an implementation of <see cref="IHttpContextAccessor" /> based on the current execution context.
    /// </summary>
    public class HttpContextAccessor : IHttpContextAccessor
    {
        private static readonly AsyncLocal<HttpContextHolder> _httpContextCurrent = new AsyncLocal<HttpContextHolder>();

        /// <inheritdoc/>
        public HttpContext? HttpContext
        {
            get
            {
                return  _httpContextCurrent.Value?.Context;
            }
            set
            {
                var holder = _httpContextCurrent.Value;
                if (holder != null)
                {
                    // Clear current HttpContext trapped in the AsyncLocals, as its done.
                    holder.Context = null;
                }

                if (value != null)
                {
                    // Use an object indirection to hold the HttpContext in the AsyncLocal,
                    // so it can be cleared in all ExecutionContexts when its cleared.
                    _httpContextCurrent.Value = new HttpContextHolder { Context = value };
                }
            }
        }

        private class HttpContextHolder
        {
            public HttpContext? Context;
        }
    }

So, it is unclear how this default implementation is consumed by ASP.NET pipeline...
Well... the questions is "How is the setter of HttpContext property consumed?"
So we have a static AsyncLocal instance... which is wrapper for HttpContextHolder
Probably AsyncLocal keeps scoped version of HttpContext but I am not sure...

[raman-m]

Additionally, I have utilized the official MS DI-helper: AddHttpContextAccessor, which also registers a singleton:

        public static IServiceCollection AddHttpContextAccessor(this IServiceCollection services)
        {
            if (services == null)
            {
                throw new ArgumentNullException(nameof(services));
            }

            services.TryAddSingleton<IHttpContextAccessor, HttpContextAccessor>();
            return services;
        }

However, upon reviewing the documentation, it seems logical that this Singleton design should yield a scoped version of the object, correct? Yet, ASP.NET applications operate correctly.
This design remains a bit perplexing to me, but it does function within the ASP.NET pipeline.

[raman-m]

@ggnaegi

Are we sure it's sound in case of multiplexing too?

No, not quite sure.
What should I do here in your opinion? 'cause I am confused after this code research
I can write some tests for scenario with aggregated routes, should I? But it is a bit beyond the scope.
What I can do right now is submitting one commit with code enhancement in .

[raman-m]

I discovered an intriguing class: HttpDataRepository in the infrastructure, which employs the same concept of sharing data across all scoped services by storing it in HttpContext.Items. The dilemma then arises: should one inject the native ASP.NET service IHttpContextAccessor, or opt for Ocelot's own IRequestScopedDataRepository? 😃

Update 1

Injecting a data repository is a strict case; therefore, injecting HttpContext is preferable as it provides extensive functionality based on the context. Consequently, the builder class can theoretically act as middleware.

[raman-m]

@ggnaegi FYI, my commit for code review is 8acd287. Honestly, I'm quite exhausted. Could you approve it? If not, I might have to pass this task over to you. 😉
Just a heads up, the configuration isn't actually consumed in the default service builder; it's set up for descendants and intended for future use. You'll find further explanations in the commit as TODO notes.

[raman-m]

TODO

• Refactor ConsulProviderFactory according to the ToDo-comments in src. Main goal is making ConsulProviderFactory as a true IServiceDiscoveryProviderFactory. Inherit from the IServiceDiscoveryProviderFactory interface, and finally moving to DI-container.
• Acceptance test(s for aggregated routes to ensure the Multiplexer is functioning correctly

[ggnaegi]

@raman-m let's accept your changes and see what's coming, ok?

[raman-m]

@ggnaegi I could extend the description of the follow up ToDo task with adding a requirement to write acceptance tests for aggregated routes to ensure Multiplexer correctly functions.
So, my plan will be to add those tests for 2119 case plus Aggregation during the next phase of refactoring of ConsulProviderFactory. Sounds good?
If we detect some Consul issue in Production, we will release a hotfix. Sounds better?

Will you deploy this version 23.3.4 to your Production env? I guess you should deploy to try all new features of ver. 23.3.0

[ggnaegi]

I will test this version later this month on a staging environment. I'm unfortunately still using Ocelot 23.2.2 on production...