Report Proportional Set Size [PSS] when available instead of Resident Set Size [RSS]

[wjordan]

In applications with large amounts of memory shared across multiple processes, Resident Set Size [RSS] may not accurately capture the amount of 'real' memory being used by a process, because it counts the total amount of shared memory separately for each process.

For example, consider a large preloaded Rails application on a 10-CPU server that uses 1 GB of memory shared (via copy-on-write) across 10 Puma processes, and another 100MB of private memory unique to each process. The shared memory is counted in the RSS separately for each process, so each process reports 1.1GB as its RSS, even though the total memory usage across all processes is actually only 2GB. This causes problems when you calculate total memory usage as the sum of RSS- in this example, the total would be reported as 11GB instead of 2GB.

To fix this issue, a newer metric called Proportional Set Size [PSS] only counts the proportional usage of each shared-memory page (so the PSS for each worker in the example would be reported as 200MB). This metric is exported via /proc/<pid>/smaps (or /proc/<pid>/smaps_rollup which pre-computes sums of all mappings) in new-ish Linux kernels, and should be used when available for more accurate memory totals.

[wjordan]

Seems related to some discussion in #7 but this isn't heroku-specific- #7 (comment) suggested opening discussion of PSS as a separate issue anyway.

[schneems]

I'm closing this for now. I don't want to change the default. If you want to move forward here I think the best thing might be to fork the project, call it GetPSSMem or something maybe? I tried adding pss support at one time and the results weren't great, most people are confused by the differences. I think that RSS is closest to what most people view as "give me my memory usage" where PSS might not quite live up to the same set of expectations.

The core of this gem is quite small and most of the complex parts are just performance optimizations. Extending this gem to handle multiple types of memory doesn't buy us much in terms of re-use and in general not many people are asking for it.