Merge pull request #1808 from ThreeMammals/release/22.0

Release 22.0

Ocelot.sln

@@ -93,6 +93,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Ocelot.Samples.ServiceDisco
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Ocelot.Samples.ServiceDiscovery.DownstreamService", "samples\OcelotServiceDiscovery\DownstreamService\Ocelot.Samples.ServiceDiscovery.DownstreamService.csproj", "{E2AC741A-4120-4D59-B5E4-16382ED45E8D}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Ocelot.Testing", "test\Ocelot.Testing\Ocelot.Testing.csproj", "{AE6BCCBD-0687-4C58-B30F-4ABBC6422087}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -203,6 +205,10 @@ Global
 		{E2AC741A-4120-4D59-B5E4-16382ED45E8D}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{E2AC741A-4120-4D59-B5E4-16382ED45E8D}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{E2AC741A-4120-4D59-B5E4-16382ED45E8D}.Release|Any CPU.Build.0 = Release|Any CPU
+		{AE6BCCBD-0687-4C58-B30F-4ABBC6422087}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{AE6BCCBD-0687-4C58-B30F-4ABBC6422087}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{AE6BCCBD-0687-4C58-B30F-4ABBC6422087}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{AE6BCCBD-0687-4C58-B30F-4ABBC6422087}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -242,6 +248,7 @@ Global
 		{25C30AAA-12DD-4BA5-A53F-9271E54EBAB7} = {8FA0CBA0-0338-48EB-B37F-83CA5022237C}
 		{D37209EA-C13E-42AE-B851-A8604F1FCD0E} = {25C30AAA-12DD-4BA5-A53F-9271E54EBAB7}
 		{E2AC741A-4120-4D59-B5E4-16382ED45E8D} = {25C30AAA-12DD-4BA5-A53F-9271E54EBAB7}
+		{AE6BCCBD-0687-4C58-B30F-4ABBC6422087} = {5B401523-36DA-4491-B73A-7590A26E420B}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {21476EFF-778A-4F97-8A56-D1AF1CEC0C48}

ReleaseNotes.md

@@ -1,17 +1,33 @@
-## Upgrade to .NET 8 (version {0}) aka [.NET 8](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) release
-> Read article: [Announcing .NET 8](https://devblogs.microsoft.com/dotnet/announcing-dotnet-8/) by Gaurav Seth, on November 14th, 2023
+## October 2023 (version {0}) aka [Swiss Locomotive](https://en.wikipedia.org/wiki/SBB-CFF-FFS_Ae_6/6) release
+> Codenamed as **[Swiss Locomotive](https://www.google.com/search?q=swiss+locomotive)**
 
-### About
-We are pleased to announce to you that we can now offer the support of [.NET 8](https://dotnet.microsoft.com/en-us/download).
-But that is not all, in this release, we are adopting support of several versions of the .NET framework through [multitargeting](https://learn.microsoft.com/en-us/dotnet/standard/frameworks).
-The Ocelot distribution is now compatible with .NET **6**, **7** and **8**. :tada:
+### Focused On
+<details>
+  <summary><b>Logging feature</b>. Performance review, redesign and improvements with new best practices to log</summary>
 
-In the future, we will try to ensure the support of the [.NET SDKs](https://dotnet.microsoft.com/en-us/download/dotnet) that are still actively maintained by the .NET team and community.
-Current .NET versions in support are the following: [6, 7, 8](https://dotnet.microsoft.com/en-us/download/dotnet).
+  - Proposing a centralized `WriteLog` method for the `OcelotLogger`
+  - Factory methods for computed strings such as `string.Format` or interpolated strings
+  - Using `ILogger.IsEnabled` before calling the native `WriteLog` implementation and invoking string factory method
+</details>
+<details>
+  <summary><b>Quality of Service feature</b>. Redesign and stabilization, and it produces less log records now.</summary>
+
+  - Fixing issue with [Polly](https://www.thepollyproject.org/) Circuit Breaker not opening after max number of retries reached
+  - Removing useless log calls that could have an impact on performance
+  - Polly [lib](https://www.nuget.org/packages/Polly#versions-body-tab) reference updating to latest `8.2.0` with some code improvements
+</details>
+<details>
+  <summary>Documentation for <b>Logging</b>, <b>Request ID</b>, <b>Routing</b> and <b>Websockets</b></summary>
+
+  - [Logging](https://ocelot.readthedocs.io/en/latest/features/logging.html)
+  - [Request ID](https://ocelot.readthedocs.io/en/latest/features/requestid.html)
+  - [Routing](https://ocelot.readthedocs.io/en/latest/features/routing.html)
+  - [Websockets](https://ocelot.readthedocs.io/en/latest/features/websockets.html)
+</details>
+<details>
+  <summary>Testing improvements and stabilization aka <b>bug fixing</b></summary>
 
-### Technical info
-As an ASP.NET Core app, now Ocelot targets `net6.0`, `net7.0` and `net8.0` frameworks.
-
-Starting with **v{0}**, the solution's code base supports [Multitargeting](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-multitargeting-overview) as SDK-style projects.
-It should be easier for teams to move between (migrate to) .NET 6, 7 and 8 frameworks. Also, new features will be available for all .NET SDKs which we support via multitargeting.
-Find out more here: [Target frameworks in SDK-style projects](https://learn.microsoft.com/en-us/dotnet/standard/frameworks)
+  - [Routing](https://ocelot.readthedocs.io/en/latest/features/routing.html) bug fixing: query string placeholders including **CatchAll** one aka `{{everything}}` and query string duplicates removal
+  - [QoS](https://ocelot.readthedocs.io/en/latest/features/qualityofservice.html) bug fixing: Polly circuit breaker exceptions
+  - Testing bug fixing: rare failed builds because of unstable Polly tests. Acceptance common logic for ports
+</details>

build.cake

@@ -161,7 +161,8 @@ Task("CreateReleaseNotes")
 		var releaseHeader = string.Format(System.IO.File.ReadAllText("./ReleaseNotes.md"), releaseVersion, lastRelease);
 		releaseNotes = new List<string> { releaseHeader };
 
-		var shortlogSummary = GitHelper($"shortlog --no-merges --numbered --summary {lastRelease}..HEAD");
+		var shortlogSummary = GitHelper($"shortlog --no-merges --numbered --summary {lastRelease}..HEAD")
+			.ToList();
 		var re = new Regex(@"^[\s\t]*(?'commits'\d+)[\s\t]+(?'author'.*)$");
 		var summary = shortlogSummary
 			.Where(x => re.IsMatch(x))
@@ -207,7 +208,6 @@ Task("CreateReleaseNotes")
 		static string HonorForDeletions(string place, string author, int commits, int files, int insertions, int deletions)
 			=> HonorForInsertions(place, author, commits, files, insertions, $"and **{deletions}** deletion{Plural(deletions)}");
 
-		var statistics = new List<(string Contributor, int Files, int Insertions, int Deletions)>();
 		foreach (var group in commitsGrouping)
 		{
 			if (topContributors.Count >= top3) break;
@@ -220,6 +220,7 @@ Task("CreateReleaseNotes")
 			}
 			else // multiple candidates with the same number of commits, so, group by files changed
 			{
+				var statistics = new List<(string Contributor, int Files, int Insertions, int Deletions)>();
 				var shortstatRegex = new Regex(@"^\s*(?'files'\d+)\s+files?\s+changed(?'ins',\s+(?'insertions'\d+)\s+insertions?\(\+\))?(?'del',\s+(?'deletions'\d+)\s+deletions?\(\-\))?\s*$");
 				// Collect statistics from git log & shortlog
 				foreach (var author in group.authors)
@@ -315,15 +316,15 @@ private void WriteReleaseNotes()
 	Information($"RUN {nameof(WriteReleaseNotes)} ...");
 
 	EnsureDirectoryExists(packagesDir);
-	System.IO.File.WriteAllLines(releaseNotesFile, releaseNotes);
+	System.IO.File.WriteAllLines(releaseNotesFile, releaseNotes, Encoding.UTF8);
 
-	var content = System.IO.File.ReadAllText(releaseNotesFile);
+	var content = System.IO.File.ReadAllText(releaseNotesFile, Encoding.UTF8);
 	if (string.IsNullOrEmpty(content))
 	{
 		System.IO.File.WriteAllText(releaseNotesFile, "No commits since last release");
 	}
 
-	Information($"Release notes are >>>\n{content}<<<");
+	Information("Release notes are >>>\n{0}<<<", content);
 	Information($"EXITED {nameof(WriteReleaseNotes)}");
 }
 
@@ -510,10 +511,11 @@ Task("PublishToNuget")
     .IsDependentOn("DownloadGitHubReleaseArtifacts")
     .Does(() =>
     {
-		if (IsRunningOnCircleCI())
-		{
-			PublishPackages(packagesDir, artifactsFile, nugetFeedStableKey, nugetFeedStableUploadUrl, nugetFeedStableSymbolsUploadUrl);
-		}
+		Information("Skipping of publishing to NuGet...");
+		// if (IsRunningOnCircleCI())
+		// {
+		// 	PublishPackages(packagesDir, artifactsFile, nugetFeedStableKey, nugetFeedStableUploadUrl, nugetFeedStableSymbolsUploadUrl);
+		// }
 	});
 
 RunTarget(target);

docs/conf.py

@@ -9,7 +9,7 @@
 project = 'Ocelot'
 copyright = ' 2023 ThreeMammals Ocelot team'
 author = 'Tom Pallister, Ocelot Core team at ThreeMammals'
-release = '21.0'
+release = '22.0'
 
 # -- General configuration ---------------------------------------------------
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration

docs/features/caching.rst

@@ -30,7 +30,7 @@ Finally, in order to use caching on a route in your Route configuration add this
 
 .. code-block:: json
 
-    "FileCacheOptions": { "TtlSeconds": 15, "Region": "somename" }
+    "FileCacheOptions": { "TtlSeconds": 15, "Region": "europe-central" }
 
 In this example **TtlSeconds** is set to 15 which means the cache will expire after 15 seconds.
 The **Region** represents a region of caching. 

docs/features/configuration.rst

@@ -241,6 +241,8 @@ Use ``HttpHandlerOptions`` in a Route configuration to set up ``HttpHandler`` be
 
 * **MaxConnectionsPerServer** This controls how many connections the internal ``HttpClient`` will open. This can be set at Route or global level.
 
+.. _ssl-errors:
+
 SSL Errors
 ----------
 

docs/features/logging.rst

@@ -2,23 +2,158 @@ Logging
 =======
 
 Ocelot uses the standard logging interfaces ``ILoggerFactory`` and ``ILogger<T>`` at the moment.
-This is encapsulated in ``IOcelotLogger`` and ``IOcelotLoggerFactory`` with an implementation for the standard `ASP.NET Core logging <https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/>`_ stuff at the moment.
-This is because Ocelot adds some extra info to the logs such as **request ID** if it is configured.
+This is encapsulated in ``IOcelotLogger`` and ``IOcelotLoggerFactory`` with the implementation for the standard `ASP.NET Core logging <https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/>`_ stuff at the moment.
+This is because Ocelot adds some extra info to the logs such as **RequestId** if it is configured.
 
 There is a global `error handler middleware <https://github.com/search?q=repo%3AThreeMammals%2FOcelot%20ExceptionHandlerMiddleware&type=code>`_ that should catch any exceptions thrown and log them as errors.
 
-Finally, if logging is set to **Trace** level, Ocelot will log starting, finishing and any middlewares that throw an exception which can be quite useful.
+Finally, if logging is set to ``Trace`` level, Ocelot will log starting, finishing and any middlewares that throw an exception which can be quite useful.
+
+Request ID
+----------
 
 The reason for not just using `bog standard <https://notoneoffbritishisms.com/2015/03/27/bog-standard/>`_ framework logging is that
-we could not work out how to override the request id that get's logged when setting **IncludeScopes** to ``true`` for logging settings.
+we could not work out how to override the **RequestId** that get's logged when setting **IncludeScopes** to ``true`` for logging settings.
 Nicely onto the next feature.
 
+Every log record has these 2 properties:
+
+* **RequestId** represents ID of the current request as plain string, for example ``0HMVD33IIJRFR:00000001``
+* **PreviousRequestId** represents ID of the previous request
+
+As an ``IOcelotLogger`` interface object being injected to constructors of service classes, current default Ocelot logger (``OcelotLogger`` class) reads these 2 properties from the ``IRequestScopedDataRepository`` interface object.
+Find out more about these properties and other details on the *Request ID* logging feature in the  :doc:`../features/requestid` chapter.
+
+.. _logging-warning:
+
 Warning
 -------
 
-If you are logging to `Console <https://learn.microsoft.com/en-us/dotnet/api/system.console>`_, you will get terrible performance.
-The team has had so many issues about performance issues with Ocelot and it is always logging level **Debug**, logging to `Console <https://learn.microsoft.com/en-us/dotnet/api/system.console>`_.
+If you are logging to MS `Console <https://learn.microsoft.com/en-us/dotnet/api/system.console>`_, you will get terrible performance.
+The team has had so many issues about performance issues with Ocelot and it is always logging level ``Debug``, logging to `Console <https://learn.microsoft.com/en-us/dotnet/api/system.console>`_.
 
 * **Warning!** Make sure you are logging to something proper in production environment!
-* Use **Error** and **Critical** levels in production environment!
-* Use **Warning** level in testing environment!
+* Use ``Error`` and ``Critical`` levels in production environment!
+* Use ``Warning`` level in testing & staging environments!
+
+These and other recommendations are below in the :ref:`logging-best-practices` section.
+
+.. _logging-best-practices:
+
+Best Practices
+--------------
+
+    | Microsoft Learn сomplete reference: `Logging in .NET Core and ASP.NET Core <https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/>`_
+
+Our recommendations to gain Ocelot best logging are the following.
+
+First
+^^^^^
+
+Ensure minimum level while `Configure logging <https://learn.microsoft.com/en-us/aspnet/core/fundamentals/logging/#configure-logging>`_.
+The minimum log level is set in the application's ``appsettings.json`` file. This level is defined in the **Logging** section, for example:
+
+.. code-block:: json
+
+  {
+    "Logging": {
+      "LogLevel": {
+        "Default": "Information",
+        "Microsoft.AspNetCore": "Warning"
+      }
+    }
+  }
+
+Whether using `Serilog <https://serilog.net/>`_ or the standard Microsoft providers, the logging configuration will be retrieved from this section.
+
+.. code-block:: csharp
+
+    .ConfigureAppConfiguration((_, config) =>
+    {
+        config.AddJsonFile($"appsettings.{env.EnvironmentName}.json", false, false);
+        // ...
+    })
+
+However, there is one thing to be aware of. It is possible to use the ``SetMinimumLevel()`` method to define the minimum logging level.
+Be careful and make sure you set the log level in one place only, like:
+
+.. code-block:: csharp
+
+    ConfigureLogging(logging =>
+    {
+        logging.ClearProviders();
+        logging.SetMinimumLevel(minLogLevel);
+        logging.AddConsole(); // MS Console for Development and/or Testing environments only
+    })
+
+Please also use the ``ClearProviders()`` method, so that only the providers you wish to use are taken into account, as in the example above, the console.
+
+Second
+^^^^^^
+
+Ensure proper usage of minimum logging level for each environment: development, testing, production, etc.
+So, once again, read important notes of the :ref:`logging-warning` section! 
+
+Third
+^^^^^
+
+Ocelot's logging has been improved in `22.0 <https://github.com/ThreeMammals/Ocelot/releases/tag/22.0.0>`_ version:
+it is now possible to use a factory method for message strings that will only be executed if the minimum log level allows it.
+
+For example, let's take a message containing information about several variables that should only be generated if the minimum log level is ``Debug``.
+If the minimum log level is ``Warning`` then the string is never generated.
+
+Therefore, when the string contains dynamic information aka ``string.Format``, or string value is generated by `string interpolation <https://learn.microsoft.com/en-us/dotnet/csharp/tutorials/string-interpolation>`_ expression,
+it is recommended to call the log method using anonymous delegate via an ``=>`` expression function:
+
+.. code-block:: csharp
+
+    Logger.LogDebug(() => $"downstream templates are {string.Join(", ", response.Data.Route.DownstreamRoute.Select(r => r.DownstreamPathTemplate.Value))}");
+
+otherwise a constant string is sufficient
+
+.. code-block:: csharp
+
+    Logger.LogDebug("My const string");
+
+Performance Review
+------------------
+
+Ocelot's logging performance has been improved in version `22.0 <https://github.com/ThreeMammals/Ocelot/releases/tag/22.0.0>`__ (see PR `1745 <https://github.com/ThreeMammals/Ocelot/pull/1745>`_).
+These changes were requested as part of issue `1744 <https://github.com/ThreeMammals/Ocelot/issues/1744>`_ after team's `discussion <https://github.com/ThreeMammals/Ocelot/discussions/1736>`_.
+
+Top Logging Performance?
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here is a quick recipe for your Production environment!
+You need to ensure the minimal level is ``Critical`` or ``None``. Nothing more!
+For sure, having top logging performance means having less log records written by logging provider. So, logs should be pretty empty.
+
+Anyway, during the first time after a version release to production, we recommend to watch the system and current version app behavior by specifying ``Error`` minimum level.
+If release engineer will ensure stability of the version in production then minimum level can be increased to ``Critical`` or ``None`` to gain top performance.
+Technically this will switch off the logging feature at all.
+
+Run Benchmarks
+^^^^^^^^^^^^^^
+
+We have 2 types of benchmarks currently
+
+* ``SerilogBenchmarks`` with Serilog logging to a file. See ``ConfigureLogging`` method with ``logging.AddSerilog(_logger);``
+* ``MsLoggerBenchmarks`` with MS default logging to MS Console. See ``ConfigureLogging`` method with ``logging.AddConsole();``
+
+Benchmark results largely depend on the environment and hardware on which they run.
+We are pleased to invite you to run Logging benchmarks on your machine by the following instructions below.
+
+1. Open PowerShell or Command Prompt console
+2. Build Ocelot solution in Release mode: ``dotnet build --configuration Release``
+3. Go to ``test\Ocelot.Benchmarks\bin\Release\`` folder.
+4. Choose .NET version changing the folder, for example to ``net8.0``
+5. Run **Ocelot.Benchmarks.exe**: ``.\Ocelot.Benchmarks.exe``
+6. Run ``SerilogBenchmarks`` or ``MsLoggerBenchmarks`` by pressing appropriate number of a benchmark: ``5`` or ``6``, + Enter
+7. Wait for 3+ minutes to complete benchmark, and get final results.
+8. Read and analize your benchmark session results.
+
+Indicators
+^^^^^^^^^^
+
+``To be developed...``