Looking for clarity on Process Metrics

[TimothyMothra]

I'm looking for some clarity on the Process Metrics.
Specifically, why is "process.memory.usage" defined as "physical memory"?
https://github.com/open-telemetry/semantic-conventions/blob/main/docs/system/process-metrics.md#metric-processmemoryusage

It seems to me that "memory.usage" is vague, and it's not clear to me why this was assigned "physical memory".

I found the original PR and I can see from the conversation that this was originally:

| `process.memory.physical_usage` | Asynchronous UpDownCounter | By | The amount of physical memory in use. |  |
| `process.memory.virtual_usage` | Asynchronous UpDownCounter | By | Virtual memory size. |  |

And then and some point the PR was changed to:

| `process.memory.usage` | Asynchronous UpDownCounter | By | The amount of physical memory in use. |  |
| `process.memory.virtual` | Asynchronous UpDownCounter | By | The amount of committed virtual memory. |  |

Proposal

As this spec is still Experimental, is there appetite to make changes?
I propose for consideration:

• process.memory.physical
• process.memory.private
• process.memory.virtual

Example for Discussion

I'm a .NET developer and I'm specifically interested in the Private Bytes. This is the value that various other tools show me, but it also seems to be more representative of the total memory currently allocated.

Sample code to generate memory pressure (CLICK TO EXPAND)

long totalMemoryAllocated = 0;
int chunkSizeMB = 1000;

try
{
    while (true)
    {
        // Allocate memory in chunks
        byte[]? memoryChunk = new byte[chunkSizeMB * 1024 * 1024]; // 1 MB = 1024 * 1024 bytes
        totalMemoryAllocated += memoryChunk.Length;

        // Release the allocated memory
        memoryChunk = null;

        System.Threading.Thread.Sleep(500);
    }
}
catch (OutOfMemoryException)
{
    Console.WriteLine("Out of memory!");
}

• Visual Studio's debugger Diagnostic Tools shows me Private Bytes.
  

• Process Explorer shows me Private Bytes
  

Terminology

While investigating this, it seems that there are conflicting definitions depending on where you read.
If someone has some preferred definitions of the different types of memory, or examples from other languages please share in the comments! :)

.NET provides some metrics with their own definitions:

• Process.PrivateMemorySize64 (WHILE DEBUGGING, THIS MOST CLOSELY MATCHES MY SCREENSHOTS ABOVE)
  The amount of memory, in bytes, allocated for the associated process that cannot be shared with other processes.
  The property value is equivalent to the Private Bytes performance counter for the process.
• Process.WorkingSet64
  The amount of physical memory, in bytes, allocated for the associated process.
  The working set of a process is the set of memory pages currently visible to the process in physical RAM memory. These pages are resident and available for an application to use without triggering a page fault.
  The property value is equivalent to the Working Set performance counter for the process.
• Process.VirtualMemorySize64
  The amount of virtual memory, in bytes, allocated for the associated process.
  The operating system maps the virtual address space for each process either to pages loaded in physical memory, or to pages stored in the virtual memory paging file on disk.
  This property can be used to monitor memory usage on computers with 32-bit processors or 64-bit processors. The property value is equivalent to the Virtual Bytes performance counter for the process.

Asking ChatGPT to define (CLICK TO EXPAND)

You
Can you provide a comparison/contrast of the following types of memory: public, private, committed, and virtual

ChatGPT
Certainly! Let's compare and contrast the following types of memory: public, private, committed, and virtual memory.

1. Public Memory:

• Public memory refers to memory that is accessible and shared among multiple processes or users.
• It is commonly used in multi-user systems or environments where processes need to share data.
• Public memory facilitates communication and data exchange between different processes or users.
• Example: Shared libraries in an operating system that are accessible to multiple processes.

2. Private Memory:

• Private memory refers to memory that is allocated exclusively to a single process and cannot be accessed by other processes.
• Each process has its own private memory space, ensuring data isolation and security.
• Private memory protects the integrity of data within a process and prevents unintended access or modification by other processes.
• Example: Stack and heap memory allocated to individual processes in an operating system.

3. Committed Memory:

• Committed memory refers to the portion of virtual memory that has been allocated to a process and is backed by physical storage, such as RAM or disk space.
• When a process requests memory, the operating system commits a portion of virtual memory to fulfill that request.
• Committed memory may not necessarily be actively used; it simply indicates that the memory has been reserved for a specific process.
• Example: When a program loads into memory, the operating system commits a certain amount of virtual memory to accommodate its requirements.

4. Virtual Memory:

• Virtual memory is a memory management technique that provides an illusion of infinite memory by allowing the operating system to use both RAM and disk storage to simulate additional memory.
• It allows processes to access more memory than physically available by swapping data between RAM and disk as needed.
• Virtual memory enables efficient memory utilization and multitasking by allowing multiple processes to run simultaneously without exhausting physical memory resources.
• Example: When RAM becomes scarce, the operating system may move less frequently accessed data from RAM to disk storage, freeing up physical memory for more active processes.

Comparison:

• Public and private memory differ primarily in terms of accessibility and sharing among processes. Public memory is shared, while private memory is exclusive to individual processes.
• Committed memory is a subset of virtual memory, representing the portion that has been allocated to processes and backed by physical storage.
• Virtual memory encompasses the entire memory management system, including both physical RAM and disk storage, providing a mechanism for extending available memory beyond physical limits.

Contrast:

• Public and private memory differ in their access control mechanisms: public memory allows access to multiple processes, while private memory is restricted to individual processes.
• Committed memory specifically refers to memory that has been allocated and backed by physical storage, while virtual memory is a broader concept encompassing the entire memory management system, including both physical and virtual storage.
• Public and private memory deal with the organization of memory within processes, while committed and virtual memory focus on the management and allocation of memory resources by the operating system.

[lmolkova]

tagging the owners - @open-telemetry/semconv-system-approvers, could you please take a look?

[TimothyMothra]

I'm continuing to investigate what metrics are available in other languages and it seems that "physical bytes" is consistently available.

.NET

• PrivateMemory, for private bytes
• WorkingSet, for physical bytes
• VirtualMemory, for virtual bytes
  https://learn.microsoft.com/en-us/dotnet/api/system.diagnostics.process

Python

• RSS: "Resident Set Size", for physical bytes
• VMS: "Virtual Memory Size”, for virtual bytes
  https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info

Node

• RSS: "Resident Set Size", for physical bytes
  https://nodejs.org/api/process.html#processmemoryusage

[mx-psi]

Discussed on the 2024-12-19 meeting. We feel like the current names make sense and are reasonably platform agnostic. We may add support for reporting the working set or private memory in the future as independent metrics, possibly Windows-specific (tbd).

Removing from GA blockers since we don't intend to make changes on the existing metrics and we do not consider this a 'core' metric to have available.