Support for cached memory statistics

[giampaolo]

From [email protected] on May 12, 2010 18:05:18

psutil has methods to check total/available/used physical memory. but it
does not have method for getting how much of the used memory is in
buffers/cache.

Original issue: http://code.google.com/p/psutil/issues/detail?id=88

[giampaolo]

From [email protected] on May 12, 2010 09:27:28

Summary: Support for cached memory statistics
Labels: -Type-Defect Type-Enhancement Milestone-0.1.4 OpSys-All Progress-0in4

[giampaolo]

From [email protected] on May 12, 2010 09:30:30

Thanks for the suggestion Kevin, we'll look at this for the next release.

Status: Accepted

[giampaolo]

From [email protected] on May 14, 2010 06:10:06

Thanks for checking on this. Any idea when the next release 
will be thanks? Almost all the time in Linux machines 
available RAM is 0 as most of it is cached. It will be great to 
have this stat.

[giampaolo]

From [email protected] on May 14, 2010 07:57:23

Not sure when the next release will be yet... are you only needing this at the moment for Linux systems? If so, 
Linux is the easiest to implement this for and we can get that checked into SVN for you real quick so you can at 
least use this feature.

[giampaolo]

From [email protected] on May 14, 2010 08:10:50

Implemented for Linux as r550 and added two new module-level methods cached_mem() and cached_swap() to 
return cached memory and swap values, in bytes. Other platforms currently raise NotImplementedError

Labels: -Progress-0in4 Progress-1in4

[giampaolo]

From g.rodola on May 14, 2010 10:57:52

I think adding two brand new methods for this is kinda "wasted".
IMHO it would be a lot better if we could include this information in
get_memory_info() and make it return a namedtuple instead of a tuple.

>>> from collections import namedtuple
>>> meminfo = namedtuple('meminfo', 'rss vms cached cachedswap')
>>> meminfo(1,2,3,4)
meminfo(rss=1, vms=2, cached=3, cachedswap=4)

This would also be backward compatible with the current implementation as a named
tuple object can be accessed by name and by position index:

>>> meminfo[0]
1
>>> meminfo.rss
1

[giampaolo]

From [email protected] on May 14, 2010 12:38:28

get_memory_info() is for per-process statistics... the cached_mem() and cached_swap() are for system-wide 
statistics, same as avail_phymem()/used_phymem() and total_virtmem()/avail_virtmem()

That's why I created new methods, for consistency with the other module-level system wide memory info.

[giampaolo]

From g.rodola on May 14, 2010 12:48:54

Oh right sorry, I didn't notice that.

[giampaolo]

From [email protected] on May 14, 2010 19:55:05

jloden
thanks a lot! works really well, and this is exactly what i want!

is it a good idea for you to add a method for this

psutil.TOTAL_PHYMEM-psutil.used_phymem()+psutil.cached_mem()

which free command reports as +-buffers/cache. this is the actual free mem.

[giampaolo]

From [email protected] on May 14, 2010 20:09:53

That's a possibility... the thought had occurred to me as well to create a "used - cache" shortcut method. I 
don't want there to be any confusion regarding free memory versus cached etc. when calling methods so if we 
do create a specialized function we'll need to make sure it's obvious/intuitive what it's for and that it's well 
documented. By the time we get it into the next official release we'll iron that out. 

On another note, I've run into some trouble with this feature on OS X - it doesn't seem like OS X has a cached 
memory exactly analogous to what we have on Linux. Here the system monitor features Free, Wired, Active and 
Inactive memory and I'm not sure what the distinctions are. More research will be needed to determine what 
the various metrics represent and if any of them map to cached memory. "Inactive" memory may be allocated 
memory not also in use which would be reasonable to call "cache" but I'll need to confirm.

FreeBSD/Windows platforms also need to be researched. I believe there is something suitable at least on the 
Windows side but it's been too long since I last read the MSDN docs to recall with certainty. Giampaolo, if you 
have some time feel free to look into this also and update the issue here or catch me on IM if you find anything 
relevant and we can look at implementing this for the other platforms. I did say Linux was the easy one ;)

Status: Started

[giampaolo]

From [email protected] on May 14, 2010 21:40:43

thanks for clarifying!!!

cached_swap is this the swap used?

i get a value which i cannot correspond to any output with free command?

[giampaolo]

From [email protected] on May 14, 2010 22:15:44

cached_swap is pulling the SwapCached value from /proc/meminfo  - more info on what this value represents can 
be found here: http://kerneltrap.org/node/4097 From what I can tell 
it doesn't look like this comes into play all that often and isn't as important as the difference 
between cached/free physical memory.

[giampaolo]

From g.rodola on June 03, 2010 13:29:30

Committed for FreeBSD as r552 which also include tests.
Maybe I'm wrong but apparently on FreeBSD there's no such thing as "cached swap":

[root@freebsd ~/psutil]# freecolor -m -o
             total       used       free     shared    buffers     cached
Mem:           499         52        446          0          0          0
Swap:          344          0        344
Segmentation fault: 11
[root@freebsd ~/psutil]# sysctl -a | grep vm. | grep swap
vm.swap_enabled: 1
vm.nswapdev: 1
vm.swap_async_max: 4
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swapin: 0
vm.disable_swapspace_pageouts: 0
vm.defer_swapspace_pageouts: 0
vm.swap_idle_enabled: 0

Maybe it goes under a different name? Not sure. As for now I just made
psutil.cached_swap() return 0.

Labels: -Progress-1in4 Progress-2in4

[giampaolo]

From g.rodola on June 22, 2010 12:37:55

Labels: -Milestone-0.1.4 Milestone-0.2.0

[giampaolo]

From [email protected] on June 22, 2010 12:38:22

More info for OS X: http://support.apple.com/kb/HT1342 -----

_Wired memory_
Information in this memory can't be moved to the hard disk, so it 
must stay in RAM. The amount of Wired memory depends on the applications you are using.

_Active memory_
This information is currently in memory, and has been recently used.

_Inactive memory_
This information in memory is not actively being used, but was recently used.

For example, if you've been using Mail and then quit it, the RAM that 
Mail was using is marked as Inactive memory. This Inactive memory is 
available for use by another application, just like Free memory.  
However, if you open Mail before its Inactive memory is used by a 
different application, Mail will open quicker because its Inactive 
memory is converted to Active memory, instead of loading Mail from the slower hard disk.

-----

Still need some research on Windows and to confirm on BSD but it looks like buffer/cache is only relevant on Linux.

[giampaolo]

From g.rodola on July 06, 2010 15:38:13

If that is relevant on Linux only I vote for not providing the functionality at all.

[giampaolo]

From [email protected] on July 06, 2010 15:51:55

We need to provide buffer/cache stats on Linux though, since physical 
memory used for buffer/cache is still available memory. If we only 
use the "free" memory stat it will always seem like there's no free 
RAM on the system even if there is lots of available phys mem.  I 
think for that reason we need to at least provide buffer/cache stats 
for Linux in case someone wants more accurate physical memory information *excluding* cache.

[giampaolo]

From [email protected] on August 05, 2010 16:57:12

We should probably make this Linux and BSD only, since it doesn't 
appear relevant to Windows and OS X. Let's update the test cases and close this out.

[giampaolo]

From g.rodola on August 05, 2010 17:43:15

Now that I'm re-reading the issue history and looking at the commits I'm a little confused.
The original request was for buffers/cache memory stats but, at least 
on Linux, we're currently providing "cache" and "cache-swap" stats, while "buffers" is left out.

I believe what the OP meant to have in psutil were "buffers" and "cached" columns as shown by free command line utility:

             total       used       free     shared    buffers     cached
Mem:       4121144    2493672    1627472          0     719200     455560
-/+ buffers/cache:    1318912    2802232
Swap:      2047992          0    2047992

[giampaolo]

From [email protected] on August 05, 2010 19:19:50

I only implemented this for cached memory, and did not include 
buffers. We could also include buffer but I'm not sure if it's needed, usually cache is the big culprit for "hiding" free memory.

[giampaolo]

From g.rodola on August 10, 2010 08:58:22

Owner: jloden

[giampaolo]

From g.rodola on September 20, 2010 15:30:52

Jay, if you agree I'd be for removing this functionality as long as 
we figure out what to do with it so that we can stop delaying 0.2.0 version.
The remaining release blocker would be issue 88 at this point.

[giampaolo]

From [email protected] on September 20, 2010 15:37:10

Are you talking about removing the cached swap statistic, or all 
cached memory statistics? Linux physical memory usage is really 
meaningless without accounting for the cache. I'm fine with only 
including this statistic on Linux, or making it an option to the 
avail_phymem() / used_phymem() methods, but I don't favor removing it because it really is needed at least on Linux platforms.

[giampaolo]

From g.rodola on September 20, 2010 16:08:41

My concern is about the delay of the new release because of this 
issue because at this point it seems it's still not clear what to do 
on other platforms and what's the best API to provide for this functionality.

If and when everything will be properly figured out I have no problems on having cached stats on Linux only.

[giampaolo]

From [email protected] on September 20, 2010 16:16:55

I don't want to leave the ability to query cached physical memory out 
of the release for Linux users. That being said, I see no reason to 
block the release on this - why not just leave it as is for now, and 
considered subject to change. We'll figure out a permanent solution 
later, and in the meantime we're not leaving our Linux users out in the cold.

[giampaolo]

From g.rodola on September 20, 2010 16:50:47

It's fine with me. In this case I think we need to make some adjustments to r550 .

- provided cached statistics should reflect results provided by free 
cmdline utility, which are "buffers" and "cached" columns. I see no 
reasons to provide other stats, hence I'd be for removing psutil.cached_swap()

- for conformity with exising API psutil.cached_mem(), which reflects 
the "cached" column of free cmdline utility, should be renamed to cached_phymem()

- free cmdline utility "buffers" column can be provided as psutil.phymem_buffers()

- because such stats don't make sense on FreeBSD, Windows and OSX, on 
such platforms cached_phymem() and psutil.phymem_buffers() should 
raise AttributeError instead of NotImplementedError. The same 
approach is used, for example, in os module: those functions which 
are not implemented on Windows are simply not included in os.* namespace.

- docstrings of both functions should specify that the returned values match free "buffers" and "cached" columns.


To summarize, the new API after this change should look like this:


psutil.TOTAL_PHYMEM
psutil.avail_phymem()
psutil.used_phymem()
psutil.cached_phymem()  (New in 0.2.0 - Linux only)
psutil.phymem_buffers()  (New in 0.2.0 - Linux only)

psutil.total_virtmem()
psutil.avail_virtmem()
psutil.used_virtmem()

[giampaolo]

From [email protected] on September 20, 2010 16:54:55

Works for me!

[giampaolo]

From g.rodola on September 23, 2010 15:04:48

Implemented in r642 .
Closing out as fixed.

Status: Fixed
Labels: -Progress-2in4 OpSys-Linux

[giampaolo]

From g.rodola on June 09, 2011 15:33:51

Labels: -OpSys-All

[giampaolo]

From g.rodola on March 02, 2013 03:52:51

Updated csets after the SVN -> Mercurial migration: r550 == revision 
9d62e8a3c7f7 r552 == revision 4939190147fb r642 == revision 8ac9d73463fd