Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Docker plugin] KeyError: 'mem' cause "Exception in ASGI application" #3091

Open
janusn opened this issue Jan 23, 2025 · 4 comments
Open

[Docker plugin] KeyError: 'mem' cause "Exception in ASGI application" #3091

janusn opened this issue Jan 23, 2025 · 4 comments
Labels
bug needs test plugin/containers
Milestone
Glances 4.3.1

Comments

@janusn
Copy link

janusn commented Jan 23, 2025 •
edited
Loading

Check the bug
Before filling this bug report, please search if a similar issue already exists.
In this case, just add a comment on this existing issue.

Describe the bug
A clear and concise description of what the bug is.

Glances logged the following errors repeatly until the container was restarted. Meanwhile, the CPU usage hopped to near 100% (of a single core)

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 260, in __update_plugin
    self._plugins[p].update_views()
  File "/app/glances/plugins/containers/__init__.py", line 289, in update_views
    self.views[i[self.get_key()]]['mem']['decoration'] = alert
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'mem'
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 260, in __update_plugin
    self._plugins[p].update_views()
  File "/app/glances/plugins/containers/__init__.py", line 289, in update_views
    self.views[i[self.get_key()]]['mem']['decoration'] = alert
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'mem'

To Reproduce
Steps to reproduce the behavior:

  1. Start container Glances with the following environment variables
    GLANCES_OPT=-C /config/glances.conf -w --password -u ha
  2. Let it run for a while
  3. The above errors were logged.

Expected behavior
A clear and concise description of what you expected to happen.
No errors logged and low CPU usage.

Screenshots
If applicable, add screenshots to help explain your problem.

Environement (please complete the following information)

  • Operating System (lsb_release -a or OS name/version): reported from the webui of glances: rockerAlpine Linux 3.21.0 64bit / Linux 6.6.44-production+truenas (lsb_release is not available inside the container)
  • Glances & psutil versions: To be completed with result of: glances -V
Glances version:	4.3.0.7
Glances API version:	4
PsUtil version:		6.1.1
Log file:		/tmp/glances-root.log
  • How do you install Glances (Pypi package, script, package manager, source): To be completed docker compose
  • Glances test: To be completed with result of: glances --issue
===============================================================================
Glances 4.3.0.7 (/app/glances/__init__.py)
Python 3.12.8 (/venv/bin/python3)
PsUtil 6.1.1 (/venv/lib/python3.12/site-packages/psutil/__init__.py)
===============================================================================
alert         [OK]    0.00002s 
[]
amps          [OK]    0.00032s key=name 
[{'count': 0,
  'countmax': None,
  'countmin': 1.0,
  'key': 'name',
  'name': 'Dropbox',
  'refresh': 3.0,
  'regex': True,
  'result': None,
  'timer': 3.8303349018096924}, ...]
cloud         [NA]
connections   [NA]
containers    [OK]    0.27914s key=name 
[{'command': 'bash -c python3 -u launcher.py',
  'cpu_percent': 2421.7516346153852,
  'created': '2025-01-20T16:09:39.387642691Z',
  'engine': 'docker',
  'id': '6a2073a187b2fc9445ca404a63e539ec2d0b19222d34dccf8c2d1141b7ed3b8e',
  'image': ('mccloud/subgen:cpu',),
  'io_rx': None,
  'io_wx': None,
  'key': 'name',
  'memory_percent': None,
  'memory_usage': 4902707200,
  'name': 'subgen-eng',
  'network_rx': None,
  'network_tx': None,
  'status': 'running',
  'uptime': '3 hours'}, ...]
core          [OK]    0.00674s 
{'log': 48, 'phys': 24}
cpu           [OK]    0.00178s 
{'cpucore': 48,
 'ctx_switches': 0,
 'ctx_switches_gauge': 133773552549,
 'guest': 0.0,
 'idle': 43.7,
 'interrupts': 0,
 'interrupts_gauge': 98311431652,
 'iowait': 0.5,
 'irq': 0.0,
 'nice': 0.1,
 'soft_interrupts': 0,
 'soft_interrupts_gauge': 32451253621,
 'steal': 0.0,
 'syscalls': 0,
 'syscalls_gauge': 0,
 'system': 3.8,
 'time_since_update': 2.7389793395996094,
 'total': 55.8,
 'user': 51.8}
diskio        [OK]    0.00767s key=disk_name 
[{'disk_name': 'nvme0n1',
  'key': 'disk_name',
  'read_bytes': 0,
  'read_bytes_gauge': 187596130304,
  'read_count': 0,
  'read_count_gauge': 8477506,
  'time_since_update': 2.6552414894104004,
  'write_bytes': 0,
  'write_bytes_gauge': 573123387392,
  'write_count': 0,
  'write_count_gauge': 43582441}, ...]
folders       [OK]    0.00005s 
[]
fs            [OK]    0.00136s key=mnt_point 
[{'device_name': 'tank/docker',
  'free': 131679122620416,
  'fs_type': 'zfs',
  'key': 'mnt_point',
  'mnt_point': '/config',
  'percent': 0.0,
  'size': 131731677642752,
  'used': 52555022336}, ...]
gpu           [OK]    0.00006s 
[]
help          [OK]    0.00001s 
None
ip            [OK]    0.00004s 
{}
irq           [NA]
load          [OK]    0.00006s 
{'cpucore': 48, 'min1': 26.65234375, 'min15': 24.41650390625, 'min5': 25.9365234375}
mem           [OK]    0.00036s 
{'active': 1332334592,
 'available': 170209140736,
 'buffers': 573440,
 'cached': 8324603904,
 'free': 170209140736,
 'inactive': 15178854400,
 'percent': 37.0,
 'shared': 246968320,
 'total': 270106341376,
 'used': 99897200640}
memswap       [OK]    0.00065s 
{'free': 0, 'percent': 0.0, 'sin': 0, 'sout': 0, 'time_since_update': 2.6865971088409424, 'total': 0, 'used': 0}
network       [OK]    0.01251s key=interface_name 
[{'alias': None,
  'bytes_all': 0,
  'bytes_all_gauge': 12840692,
  'bytes_all_rate_per_sec': 0.0,
  'bytes_recv': 0,
  'bytes_recv_gauge': 6420346,
  'bytes_recv_rate_per_sec': 0.0,
  'bytes_sent': 0,
  'bytes_sent_gauge': 6420346,
  'bytes_sent_rate_per_sec': 0.0,
  'interface_name': 'lo',
  'key': 'interface_name',
  'speed': 0,
  'time_since_update': 2.320185422897339}, ...]
now           [OK]    0.00005s 
{'custom': '2025-01-23 16:30:04 GMT', 'iso': '2025-01-23T16:30:04+00:00'}
percpu        [OK]    0.00340s key=cpu_number 
[{'cpu_number': 0,
  'dpc': None,
  'guest': 0.0,
  'guest_nice': 0.0,
  'idle': 30.4,
  'interrupt': None,
  'iowait': 0.0,
  'irq': 0.0,
  'key': 'cpu_number',
  'nice': 0.0,
  'softirq': 0.0,
  'steal': 0.0,
  'system': 4.0,
  'total': 69.6,
  'user': 65.6}, ...]
ports         [OK]    0.00001s 
[]
processcount  [OK]    0.16039s 
{'pid_max': 0, 'running': 2, 'sleeping': 363, 'thread': 2665, 'total': 378}
processlist   [OK]    0.00027s 
[]
programlist   [OK]    0.00604s 
[]
psutilversion [OK]    0.00004s 
'6.1.1'
quicklook     [OK]    0.00105s 
{'cpu': 55.8,
 'cpu_hz': 3000000000.0,
 'cpu_hz_current': 1426151833.3333333,
 'cpu_log_core': 48,
 'cpu_name': 'Intel(R) Xeon(R) Silver 4116 CPU @ 2.10GHz',
 'cpu_phys_core': 24,
 'load': 50.9,
 'mem': 37.0,
 'percpu': [{...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...}, {...},
            {...}, {...}, {...}],
 'swap': 0.0}
raid          [NA]
sensors       [OK]    0.00001s key=label 
[{'critical': 79,
  'key': 'label',
  'label': 'Composite',
  'type': <SensorType.CPU_TEMP: 'temperature_core'>,
  'unit': 'C',
  'value': 39,
  'warning': 74}, ...]
smart         [NA]
system        [OK]    0.00001s 
{'hostname': 'rocker',
 'hr_name': 'Alpine Linux 3.21.0 64bit / Linux 6.6.44-production+truenas',
 'linux_distro': 'Alpine Linux 3.21.0',
 'os_name': 'Linux',
 'os_version': '6.6.44-production+truenas',
 'platform': '64bit'}
uptime        [OK]    0.00041s 
{'seconds': 3192025}
version       [OK]    0.00003s 
'4.3.0.7'
vms           [NA]
wifi          [OK]    0.00003s 
[]
===============================================================================
Total time to update all stats: 0.48889s
===============================================================================

Additional context
Add any other context about the problem here.

You can also pastebin:
@nicolargo nicolargo added the bug label Jan 24, 2025
@nicolargo nicolargo added this to the Glances 4.3.1 milestone Jan 24, 2025
@nicolargo nicolargo changed the title Exception in ASGI application [Docker plugin] KeyError: 'mem' cause "Exception in ASGI application" Jan 24, 2025
nicolargo added a commit that referenced this issue Jan 25, 2025
@nicolargo
[Docker plugin] KeyError: 'mem' cause 'Exception in ASGI application' #…
4a4cb13 
…3091
@nicolargo
Copy link
Owner

It's strange because the variable is initialize at the begin of the loop (here https://github.com/nicolargo/glances/blob/v4.3.0.7/glances/plugins/containers/__init__.py#L266)...

Nevertheless i add an extra test before filing the 'mem' item.

@janusn Can you test the develop version ?

pip install --user -i https://test.pypi.org/simple/ Glances

@janusn
Copy link
Author

janusn commented Jan 25, 2025

@nicolargo Thanks for looking it up.

As I installed via docker, I merged the activities 0c6c4886275be342abf083823963b8a85f60e9d9 and 4a4cb135da43cd1954a3376d692d39685c50777b into my container instead.

Before the change was applied, I have noticed additional errors on the log at the frequency around once every 4 hours:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 258, in __update_plugin
    self._plugins[p].update()
  File "/app/glances/plugins/plugin/model.py", line 1124, in wrapper
    ret = fct(self, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/plugin/model.py", line 1141, in wrapper
    ret = fct(*args, **kw)
          ^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/processcount/__init__.py", line 85, in update
    glances_processes.update()
  File "/app/glances/processes.py", line 601, in update
    processlist = self.update_list(processlist)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/processes.py", line 628, in update_list
    return list_of_namedtuple_to_list_of_dict(processlist)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 495, in list_of_namedtuple_to_list_of_dict
    return [namedtuple_to_dict(d) for d in data]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 490, in namedtuple_to_dict
    return {k: (v._asdict() if hasattr(v, '_asdict') else v) for k, v in data.items()}
                                                                         ^^^^^^^^^^^^
RuntimeError: dictionary changed size during iteration
RuntimeError: dictionary changed size during iteration
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 258, in __update_plugin
    self._plugins[p].update()
  File "/app/glances/plugins/plugin/model.py", line 1124, in wrapper
    ret = fct(self, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/plugin/model.py", line 1141, in wrapper
    ret = fct(*args, **kw)
          ^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/processcount/__init__.py", line 85, in update
    glances_processes.update()
  File "/app/glances/processes.py", line 601, in update
    processlist = self.update_list(processlist)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/processes.py", line 628, in update_list
    return list_of_namedtuple_to_list_of_dict(processlist)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 495, in list_of_namedtuple_to_list_of_dict
    return [namedtuple_to_dict(d) for d in data]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 490, in namedtuple_to_dict
    return {k: (v._asdict() if hasattr(v, '_asdict') else v) for k, v in data.items()}
                                                                         ^^^^^^^^^^^^
RuntimeError: dictionary changed size during iteration

@andreyryabtsev andreyryabtsev mentioned this issue Jan 28, 2025
Open
@janusn
Copy link
Author

janusn commented Feb 2, 2025 •
edited
Loading

Hi there.

After develop version was applied, an error was logged again. Here is one of the logs:

INFO:     Started server process [995]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:61208 (Press CTRL+C to quit)
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 260, in __update_plugin
    self._plugins[p].update_views()
  File "/app/glances/plugins/containers/__init__.py", line 291, in update_views
    self.views[i[self.get_key()]]['mem']['decoration'] = alert
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'mem'

@janusn
Copy link
Author

janusn commented Feb 4, 2025

Here is the errors logged in another container:

ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2505, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 1005, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/outputs/glances_restful_api.py", line 464, in _api_all
    self.__update_stats()
  File "/app/glances/outputs/glances_restful_api.py", line 186, in __update_stats
    self.stats.update()
  File "/app/glances/stats.py", line 269, in update
    self.__update_plugin(p)
  File "/app/glances/stats.py", line 258, in __update_plugin
    self._plugins[p].update()
  File "/app/glances/plugins/plugin/model.py", line 1124, in wrapper
    ret = fct(self, *args, **kw)
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/plugin/model.py", line 1141, in wrapper
    ret = fct(*args, **kw)
          ^^^^^^^^^^^^^^^^
  File "/app/glances/plugins/processcount/__init__.py", line 85, in update
    glances_processes.update()
  File "/app/glances/processes.py", line 601, in update
    processlist = self.update_list(processlist)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/processes.py", line 628, in update_list
    return list_of_namedtuple_to_list_of_dict(processlist)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 495, in list_of_namedtuple_to_list_of_dict
    return [namedtuple_to_dict(d) for d in data]
            ^^^^^^^^^^^^^^^^^^^^^
  File "/app/glances/globals.py", line 490, in namedtuple_to_dict
    return {k: (v._asdict() if hasattr(v, '_asdict') else v) for k, v in data.items()}
                                                                         ^^^^^^^^^^^^
RuntimeError: dictionary changed size during iteration

The frequency of errors was around twice a day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug needs test plugin/containers
Projects
None yet
Milestone
Glances 4.3.1
Development

No branches or pull requests
2 participants
@nicolargo @janusn