[NetBSD] EBUSY on cmdline() #2250

0-wiz-0 · 2023-04-29T21:14:20Z

I tried using bpytop on NetBSD. Usually it works fine, but sometimes, especially when there's high load, the cpu frequency backend returns "OSerror [Errno 16]: Device busy" and bpytop exits.

From the bpytop error log:

22/04/23 (00:49:54) | ERROR: Exception while getting cpu frequency!
22/04/23 (00:49:54) | ERROR: module 'psutil' has no attribute 'cpu_freq'
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.11/site-packages/bpytop.py", line 3080, in _collect
    if CONFIG.show_cpu_freq and hasattr(psutil.cpu_freq(), "current"):
                                        ^^^^^^^^^^^^^^^
AttributeError: module 'psutil' has no attribute 'cpu_freq'
22/04/23 (08:28:47) | ERROR: Data collection thread failed with exception: [Errno 16] Device busy
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.11/site-packages/bpytop.py", line 2958, in _runner
    collector._collect()
  File "/usr/pkg/lib/python3.11/site-packages/bpytop.py", line 3728, in _collect
    for p in sorted(psutil.process_iter(cls.p_values + (["memory_info"] if CONFIG.proc_mem_bytes else []), err), key=lambda p: eval(sort_cmd), reverse=reverse):
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 1445, in process_iter
    yield add(pid)
          ^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 1426, in add
    proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 528, in as_dict
    ret = meth()
          ^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 686, in cmdline
    return self._proc.cmdline()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/_psbsd.py", line 566, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/_psbsd.py", line 670, in cmdline
    return cext.proc_cmdline(self.pid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 16] Device busy
22/04/23 (08:28:49) | WARNING: Exiting with errorcode (1). Runtime 7:38:57

I'm not completely sure if this is a bug in bpytop or psutil (or possibly even NetBSD but I think returning EBUSY sounds fine).
Please advise if I should report this as a bpytop bug and how you suggest they handle it better there. Thanks!

The text was updated successfully, but these errors were encountered:

giampaolo · 2023-04-30T16:35:39Z

I see 2 errors:

22/04/23 (00:49:54) | ERROR: module 'psutil' has no attribute 'cpu_freq'
Traceback (most recent call last):
  File "/usr/pkg/lib/python3.11/site-packages/bpytop.py", line 3080, in _collect
    if CONFIG.show_cpu_freq and hasattr(psutil.cpu_freq(), "current"):
                                        ^^^^^^^^^^^^^^^
AttributeError: module 'psutil' has no attribute 'cpu_freq'

This is a bug in bpytop because cpu_freq() is not supported on NetBSD, so bpytop should make an hasattr check before invoking it.

The other error is:

  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 1426, in add
    proc.info = proc.as_dict(attrs=attrs, ad_value=ad_value)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 528, in as_dict
    ret = meth()
          ^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/__init__.py", line 686, in cmdline
    return self._proc.cmdline()
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/_psbsd.py", line 566, in wrapper
    return fun(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/pkg/lib/python3.11/site-packages/psutil/_psbsd.py", line 670, in cmdline
    return cext.proc_cmdline(self.pid)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 16] Device busy

This can be considered a psutil bug, even though I'm not sure how psutil should handle this case.
EBUSY looks sort of similar to EAGAIN in socket programming: when that happens, a network app is supposed to retry.
But I wonder: can EAGAIN occur again after retry? Probably yes, so I suppose we may retry a certain number of times (say 10), then giveup and raise AccessDenied.

I guess you don't know whether EBUSY can occur again, correct?

giampaolo · 2023-04-30T16:40:21Z

Also another question. What psutil version is this (python3 -c "import psutil; print(psutil.__ver__)")?
This exception:

OSError: [Errno 16] Device busy

...is missing some info. I would expect it to look like:

OSError: [Errno 16] Device busy (originated from sysctl(KERN_PROC_ARGV) get size)

...or

OSError: [Errno 16] Device busy (originated from sysctl(KERN_PROC_ARGV))

...as per source code:

psutil/psutil/arch/netbsd/proc.c

Lines 354 to 369 in 7282a92

    
           st = sysctl(mib, __arraycount(mib), NULL, &len, NULL, 0); 
        
           if (st == -1) { 
        
               PyErr_SetFromOSErrnoWithSyscall("sysctl(KERN_PROC_ARGV) get size"); 
        
               goto error; 
        
           } 
        
           procargs = (char *)malloc(len); 
        
           if (procargs == NULL) { 
        
               PyErr_NoMemory(); 
        
               goto error; 
        
           } 
        
           st = sysctl(mib, __arraycount(mib), procargs, &len, NULL, 0); 
        
           if (st == -1) { 
        
               PyErr_SetFromOSErrnoWithSyscall("sysctl(KERN_PROC_ARGV)"); 
        
               goto error; 
        
           }

giampaolo · 2023-04-30T16:47:32Z

I tried this stress test hoping to reproduce the issue but could not:

import psutil
while 1:
    for p in psutil.process_iter():
        p.cmdline()

giampaolo · 2023-04-30T16:53:41Z

I tried installing bpytop on NetBSD and at some point I got:

30/04/23 (16:50:52) | ERROR: Data collection thread failed with exception: invalid attr name 'cpu_num'
Traceback (most recent call last):
  File "/home/vagrant/.local/lib/python3.11/site-packages/bpytop.py", line 2958, in _runner
    collector._collect()
  File "/home/vagrant/.local/lib/python3.11/site-packages/bpytop.py", line 3795, in _collect
    cls.details = det.as_dict(attrs=attrs, ad_value="")
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/vagrant/psutil/psutil/__init__.py", line 515, in as_dict
    raise ValueError("invalid attr name%s %s" % (
ValueError: invalid attr name 'cpu_num'
30/04/23 (16:50:52) | WARNING: Exiting with errorcode (1). Runtime 0:00:40

@aristocratos, FYI this looks like another missing hasattr check by bpytop (see also my previous message about cpu_freq()).

aristocratos · 2023-04-30T17:22:40Z

@giampaolo
Noted, thanks.
But to be fair, bpytop only proclaims to support Linux, Macos and FreeBSD.
Not well enough versed in the different BSD flavours to know how much FreeBSD and NetBSD differ, but could possibly be other compatibility issues from code not relying on psutil in there too.

Gonna try at some point in the future to look over the code and check for any other issues for NetBSD and OpenBSD (and add hasattr checks where needed).
Haven't really put much priority on bpytop maintenance (or had much time) since the release of btop++.

aristocratos · 2023-04-30T18:41:07Z

@0-wiz-0 Feel free to open a request for NetBSD support in the bpytop repository, with a link to this issue (to remind me of the current problems).
Or a pull request with fixes, if you're up for it :)

0-wiz-0 · 2023-05-01T07:15:53Z

Thanks for the analysis!
I think the retry-on-EBUSY sounds good, but no, I don't know if or how often that can repeat.
Right now, I have 5.9.5 installed but it might have been 5.9.4 during the test from which the log is.

python3 -c "import psutil; print(psutil.__version__)"
5.9.5

giampaolo · 2024-06-02T23:49:50Z

This happened just now when running CI tests: https://github.com/giampaolo/psutil/actions/runs/9342019978/job/25709491422.

2024-06-02T23:38:46.5877794Z ======================================================================
2024-06-02T23:38:46.5884280Z ERROR: psutil.tests.test_process.TestProcess.test_long_cmdline
2024-06-02T23:38:46.5889870Z ----------------------------------------------------------------------
2024-06-02T23:38:46.5893256Z Traceback (most recent call last):
2024-06-02T23:38:46.5894359Z   File "/home/runner/work/psutil/psutil/psutil/tests/test_process.py", line 762, in test_long_cmdline
2024-06-02T23:38:46.5895477Z     self.assertEqual(p.cmdline(), cmdline)
2024-06-02T23:38:46.5897319Z                      ^^^^^^^^^^^
2024-06-02T23:38:46.5898294Z   File "/home/runner/work/psutil/psutil/psutil/__init__.py", line 724, in cmdline
2024-06-02T23:38:46.5899191Z     return self._proc.cmdline()
2024-06-02T23:38:46.5899686Z            ^^^^^^^^^^^^^^^^^^^^
2024-06-02T23:38:46.5900474Z   File "/home/runner/work/psutil/psutil/psutil/_psbsd.py", line 604, in wrapper
2024-06-02T23:38:46.5901353Z     return fun(self, *args, **kwargs)
2024-06-02T23:38:46.5901885Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-02T23:38:46.5902740Z   File "/home/runner/work/psutil/psutil/psutil/_psbsd.py", line 709, in cmdline
2024-06-02T23:38:46.5903650Z     return cext.proc_cmdline(self.pid)
2024-06-02T23:38:46.5904242Z            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
2024-06-02T23:38:46.5905034Z OSError: [Errno 16] Device busy (originated from sysctl(KERN_PROC_ARGV))

...which originates from:

psutil/psutil/arch/netbsd/proc.c

Lines 362 to 366 in 20be5ae

    
           st = sysctl(mib, __arraycount(mib), procargs, &len, NULL, 0); 
        
           if (st == -1) { 
        
               PyErr_SetFromOSErrnoWithSyscall("sysctl(KERN_PROC_ARGV)"); 
        
               goto error; 
        
           }

`Process.cmdline()` sometimes fail with EBUSY. It usually happens for long cmdlines with lots of arguments. In this case retry getting the cmdline for up to 50 times, and return an empty list as last resort.

github-actions bot added the bug label Apr 29, 2023

giampaolo changed the title ~~data collection failure (due to EBUSY) breaks downstreams~~ [NetBSD] EBUSY) on cmdline() Apr 30, 2023

giampaolo changed the title ~~[NetBSD] EBUSY) on cmdline()~~ [NetBSD] EBUSY on cmdline() Apr 30, 2023

0-wiz-0 mentioned this issue May 1, 2023

Add support for NetBSD aristocratos/bpytop#399

Open

giampaolo added a commit that referenced this issue May 2, 2023

NetBSD, cmdline(): add debug message on EINVAL (re. to #2250)

7290661

giampaolo mentioned this issue Jun 1, 2023

[CentOS] cpu_freq OSError: [Errno 16] Device or resource busy #2261

Open

giampaolo mentioned this issue Jun 4, 2024

NetBSD / cmdline: retry on EBUSY (fix #2250) #2421

Merged

giampaolo closed this as completed in #2421 Jun 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[NetBSD] EBUSY on cmdline() #2250

[NetBSD] EBUSY on cmdline() #2250

0-wiz-0 commented Apr 29, 2023

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading

aristocratos commented Apr 30, 2023

aristocratos commented Apr 30, 2023

0-wiz-0 commented May 1, 2023

giampaolo commented Jun 2, 2024

[NetBSD] EBUSY on cmdline() #2250

[NetBSD] EBUSY on cmdline() #2250

Comments

0-wiz-0 commented Apr 29, 2023

giampaolo commented Apr 30, 2023 • edited Loading

giampaolo commented Apr 30, 2023 • edited Loading

giampaolo commented Apr 30, 2023 • edited Loading

giampaolo commented Apr 30, 2023 • edited Loading

aristocratos commented Apr 30, 2023

aristocratos commented Apr 30, 2023

0-wiz-0 commented May 1, 2023

giampaolo commented Jun 2, 2024

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading

giampaolo commented Apr 30, 2023 •

edited

Loading