Add hooks to debug OpenSSL memory allocations

[rzikm]

This ressurects #101626. CC: @wfurt.

Changes since his PR:

• Moved some code around, the managed part now lives in Interop.Crypto.
• Moved tracking to native code and exposed to C# via foreach-callback

We had several cases when users complained about large memory use. For than native it is quite difficult to figure out where the memory goes. This PR aims to make that somewhat easier.

OpenSSL provides hooks for memory function so this PR adds switch to optimally hook into that.
The only one caveat that the CRYPTO_set_mem_functions works only if called before any allocations e.g. it needs to be done very early in the process. So I end up putting into initialization process.

The simple use pattern is something like

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Net.Http;
using System.Net.Security;
using System.Reflection;
using System.Runtime.InteropServices;
// Environment variable needs to be set before launching the process, as otherwise the native layer will
// not see the environment variable
// Environment.SetEnvironmentVariable("DOTNET_OPENSSL_MEMORY_DEBUG", "1");
//
// Once enabled, the functionality can be accessed by following methods on the Interop.Crypto class:
// - GetOpenSslAllocatedMemory - Gets the total amount of memory allocated by OpenSSL
// - GetOpenSslAllocationCount - Gets the number of allocations made by OpenSSL
// - EnableTracking/DisableTracking - toggles tracking of individual (outstanding) allocations via internal dictionary.
// - ForEachTrackedAllocation - Accepts an Action<IntPtr, int, IntPtr, int> callback and calls it for each allocation performed since the last EnableTracking call. The order of allocations is not guaranteed, the functions holds an inner lock which prevents concurrent memory operations. The parameters passed to the callback are:
//   - IntPtr to the memory allocated
//   - size of the allocation
//   - IntPtr to the filename where the allocation was made
//   - line number of the allocation

// all above mentioned APIs are accessible via "private reflection"
BindingFlags flags = BindingFlags.InvokeMethod | BindingFlags.NonPublic | BindingFlags.Static;
var cryptoInterop = typeof(RandomNumberGenerator).Assembly.GetTypes().First(t => t.Name == "Crypto");

// enable tracking, this clears up any previously tracked allocations
cryptoInterop.InvokeMember("EnableMemoryTracking", flags, null, null, null);

// do some work that includes OpenSSL
HttpClient client = new HttpClient();
await client.GetAsync("https://www.microsoft.com");

// stop tracking (this step is optional)
cryptoInterop.InvokeMember("DisableMemoryTracking", flags, null, null, null);

using var process = Process.GetCurrentProcess();
Console.WriteLine($"Bytes known to GC [{GC.GetTotalMemory(false)}], process working set [{process.WorkingSet64}]");
Console.WriteLine("OpenSSL - currently allocated memory: {0} B", cryptoInterop.InvokeMember("GetOpenSslAllocatedMemory", flags, null, null, null));
Console.WriteLine("OpenSSL - total allocations since start: {0}", cryptoInterop.InvokeMember("GetOpenSslAllocationCount", flags, null, null, null));

Dictionary<(IntPtr file, int line), ulong> allAllocations = new();
Action<IntPtr, ulong, IntPtr, int> callback = (ptr, size, namePtr, line) =>
{
    CollectionsMarshal.GetValueRefOrAddDefault(allAllocations, (namePtr, line), out _) += size;
};
cryptoInterop.InvokeMember("ForEachTrackedAllocation", flags, null, null, [callback]);

Console.WriteLine("Total allocated OpenSSL memory by location");
foreach (var ((filenameptr, line), total) in allAllocations.OrderByDescending(kvp => kvp.Value).Take(10))
{
    string filename = Marshal.PtrToStringUTF8(filenameptr);
    Console.WriteLine($"{total:N0} B from {filename}:{line}");
}

Access through Reflection should be OK since this is only last resort debug hook e.g. it does not need stable API and convenient access.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[Copilot]

Copilot reviewed 2 out of 8 changed files in this pull request and generated 1 comment.

Files not reviewed (6)

• src/native/libs/System.Security.Cryptography.Native/apibridge_30.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/entrypoints.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/opensslshim.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/pal_ssl.c: Language not supported

Copilot reviewed 2 out of 10 changed files in this pull request and generated no comments.

Files not reviewed (8)

• src/libraries/System.Security.Cryptography/src/ILLink/ILLink.Descriptors.LibraryBuild.xml: Language not supported
• src/native/libs/System.Security.Cryptography.Native/CMakeLists.txt: Language not supported
• src/native/libs/System.Security.Cryptography.Native/apibridge.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/entrypoints.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/memory_debug.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/memory_debug.h: Language not supported
• src/native/libs/System.Security.Cryptography.Native/openssl.c: Language not supported
• src/native/libs/System.Security.Cryptography.Native/opensslshim.h: Language not supported

[rzikm]

/ba-g wasm failures are unrelated.