Detected blocking call to open inside the event loop (HA 2024.6)

[hmmbob]

Version of the custom_component

0.0.14

Homeassistant version

HA 2024.6.0b4

Describe the bug

Got a lot of warnings on 2024.6 beta (60+ lines) - see below

This might be related: https://developers.home-assistant.io/blog/2024/05/19/fix_zoneinfo_blocking_io/

Debug log

Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/init.py, line 158: tz=timezone("Europe/Stockholm"), (offender: /usr/local/lib/python3.12/site-packages/pytz/init.py, line 108: return open(filename, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues

Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/aio_price.py, line 110: zone = tz.gettz(zone) (offender: /usr/local/lib/python3.12/site-packages/dateutil/tz/tz.py, line 464: fileobj = open(fileobj, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues

[Hellowlol]

Thanks for report and the links 👍

[hmmbob]

Beta 5 throws even more elaborate warnings (probably due to home-assistant/core#118620 being merged for b5), pasting them here for your reference:

Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/init.py, line 158: tz=timezone("Europe/Stockholm"), (offender: /usr/local/lib/python3.12/site-packages/pytz/init.py, line 108: return open(filename, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/main.py", line 223, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 188, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component result = await _async_setup_component(hass, domain, config) File "/usr/src/homeassistant/homeassistant/setup.py", line 402, in _async_setup_component result = await task File "/config/custom_components/nordpool/init.py", line 176, in async_setup return await _dry_setup(hass, config) File "/config/custom_components/nordpool/init.py", line 158, in _dry_setup tz=timezone("Europe/Stockholm"),

Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/aio_price.py, line 110: zone = tz.gettz(zone) (offender: /usr/local/lib/python3.12/site-packages/dateutil/tz/tz.py, line 464: fileobj = open(fileobj, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/main.py", line 223, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 188, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 649, in async_add_entities await add_func(coros, entities, timeout) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 600, in _async_add_entities await coro File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 914, in _async_add_entity await entity.add_to_platform_finish() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1354, in add_to_platform_finish await self.async_added_to_hass() File "/config/custom_components/nordpool/sensor.py", line 518, in async_added_to_hass await self.handle_new_hr() File "/config/custom_components/nordpool/sensor.py", line 480, in handle_new_hr today = await self._api.today(self._area, self._currency) File "/config/custom_components/nordpool/init.py", line 110, in today return await self._someday(area, currency, "today") File "/config/custom_components/nordpool/init.py", line 99, in _someday await self.update_today(None) File "/config/custom_components/nordpool/init.py", line 80, in update_today await self._update("today") File "/config/custom_components/nordpool/init.py", line 73, in _update data = await spot.hourly(end_date=dt) File "/config/custom_components/nordpool/aio_price.py", line 228, in hourly return await self.fetch(self.HOURLY, end_date, areas) File "/usr/local/lib/python3.12/site-packages/backoff/_async.py", line 151, in retry ret = await target(*args, **kwargs) File "/config/custom_components/nordpool/aio_price.py", line 222, in fetch return join_result_for_correct_time(raw, end_date) File "/config/custom_components/nordpool/aio_price.py", line 110, in join_result_for_correct_time zone = tz.gettz(zone)

Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/aio_price.py, line 110: zone = tz.gettz(zone) (offender: /usr/local/lib/python3.12/site-packages/dateutil/tz/tz.py, line 464: fileobj = open(fileobj, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/src/homeassistant/homeassistant/main.py", line 223, in sys.exit(main()) File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main exit_code = runner.run(runtime_conf) File "/usr/src/homeassistant/homeassistant/runner.py", line 188, in run return loop.run_until_complete(setup_and_run_hass(runtime_config)) File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete self.run_forever() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever self._run_once() File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once handle._run() File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run self._context.run(self._callback, *self._args) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 649, in async_add_entities await add_func(coros, entities, timeout) File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 600, in _async_add_entities await coro File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 914, in _async_add_entity await entity.add_to_platform_finish() File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1354, in add_to_platform_finish await self.async_added_to_hass() File "/config/custom_components/nordpool/sensor.py", line 518, in async_added_to_hass await self.handle_new_hr() File "/config/custom_components/nordpool/sensor.py", line 480, in handle_new_hr today = await self._api.today(self._area, self._currency) File "/config/custom_components/nordpool/init.py", line 110, in today return await self._someday(area, currency, "today") File "/config/custom_components/nordpool/init.py", line 100, in _someday await self.update_tomorrow(None) File "/config/custom_components/nordpool/init.py", line 85, in update_tomorrow await self.update(type="tomorrow", dt=dt_utils.now() + timedelta(hours=24)) File "/config/custom_components/nordpool/init.py", line 73, in _update data = await spot.hourly(end_date=dt) File "/config/custom_components/nordpool/aio_price.py", line 228, in hourly return await self.fetch(self.HOURLY, end_date, areas) File "/usr/local/lib/python3.12/site-packages/backoff/_async.py", line 151, in retry ret = await target(*args, **kwargs) File "/config/custom_components/nordpool/aio_price.py", line 222, in fetch return join_result_for_correct_time(raw, end_date) File "/config/custom_components/nordpool/aio_price.py", line 110, in join_result_for_correct_time zone = tz.gettz(zone)

[Hellowlol]

Ill get create a fix for this this weekend.

[hmmbob]

Tx. It's just a warning, component still works fine so no worries.

[madebywitteveen]

Hello, how are you?
Can you explain how to fix this 'detected blocking' issue in Nordpool ?

had a look into init.py but don't know what to look for,

thanks

[hmmbob]

There's a link in the start post, and otherwise just wait for the next release. It's only a warning, nothing is broken.

[madebywitteveen]

Thank you, I am trying to display 36 hours on an apex chart but it doesn't work, the chart says 'loading...'
It is the code from smarthomejunkie.
Maybe you know a fix? Thanks

[PeterAndreS]

HA becomes unresposive with nordpool - using apexcharts-card in Default dashboard or openening a dashboard with noordpool in apexcharts-card:
But only when using "windows/browser" not with "Android/app" (The dashboard is rendered complete though! And then the browser window needs to be killed.)
Update: The logging is still coming but the HA is responsive today now when no tomorrow values are available.

Core 2024.6.2
Supervisor 2024.06.0
Operating System 12.3
Frontend 20240610.0
And nordpool 0.0.14

2024-06-13 21:14:14.097 WARNING (MainThread) [homeassistant.util.loop] Detected blocking call to open inside the event loop by custom integration 'nordpool' at custom_components/nordpool/init.py, line 158: tz=timezone("Europe/Stockholm"), (offender: /usr/local/lib/python3.12/site-packages/pytz/init.py, line 108: return open(filename, 'rb')), please create a bug report at https://github.com/custom-components/nordpool/issues
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/usr/src/homeassistant/homeassistant/main.py", line 223, in
sys.exit(main())
File "/usr/src/homeassistant/homeassistant/main.py", line 209, in main
exit_code = runner.run(runtime_conf)
File "/usr/src/homeassistant/homeassistant/runner.py", line 190, in run
return loop.run_until_complete(setup_and_run_hass(runtime_config))
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 672, in run_until_complete
self.run_forever()
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 639, in run_forever
self._run_once()
File "/usr/local/lib/python3.12/asyncio/base_events.py", line 1988, in _run_once
handle._run()
File "/usr/local/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 299, in async_setup_platform
platform = await async_prepare_setup_platform(
File "/usr/src/homeassistant/homeassistant/setup.py", line 538, in async_prepare_setup_platform
) and not await async_setup_component(hass, integration.domain, hass_config):
File "/usr/src/homeassistant/homeassistant/setup.py", line 165, in async_setup_component
result = await _async_setup_component(hass, domain, config)
File "/usr/src/homeassistant/homeassistant/setup.py", line 402, in _async_setup_component
result = await task
File "/config/custom_components/nordpool/init.py", line 176, in async_setup
return await _dry_setup(hass, config)
File "/config/custom_components/nordpool/init.py", line 158, in _dry_setup
tz=timezone("Europe/Stockholm"),

[sveinse]

Another effect of this issue is that it cause very high CPU loads, close to 100% load on one CPU. I was alerted of this issue when I noticed very high loads in HA which runs on a VM. After a bit of diagnostics, found it to be caused by the nordpool integration. I also have the notice in the logs "Detected blocking call to open inside the event loop by custom integration 'nordpool'".

I'm running nordpool 0.14.0 on HA 2024.6.3 on supervisor 2024.06.0.

[rrooggiieerr]

This issue might be "only a warning", but it is actually holding up HA for quite some time and clogging up the logs with these warning messages.

Hereby a link to the HA developer blog abut the issue and how to fix it:
https://developers.home-assistant.io/blog/2024/05/19/fix_zoneinfo_blocking_io

[rrooggiieerr]

I created this pull request which looks up the timezone without blocking the event loop. There are some other blocking calls though, so this doesn't fix everything.

#383

[hmmbob]

Confirming that #383 fixes the issues.

For those who wish to try as well: you need to install the master branch of the Nordpool component.

Go to developer tools -> services and run the update.install service:

service: update.install
target:
  entity_id: update.nordpool_update
data:
  version: master

[darkrain-nl]

Confirming that #383 fixes the issues.

For those who wish to try as well: you need to install the master branch of the Nordpool component.

Go to developer tools -> services and run the update.install service:

service: update.install
target:
  entity_id: update.nordpool_update
data:
  version: master

I wanted to try but don't have update.nordpool_update so I'll wait for it to be released...

[hmmbob]

I wanted to try but don't have update.nordpool_update so I'll wait for it to be released...

Weird, it should be there if you installed through HACS?

If you did install manually, just download the master branch and overwrite the files in custom_components\nordpool.

edit: thinking of it, it could possible be an experimental feature of HACS (you can enable that in HACS)
edit2: confirmed, it's an experimental feature of HACS: https://hacs.xyz/docs/entities/update_entities

You can enable that in HACS configuration: go to your integrations page, click 'HACS' and click 'configure', it's the last checkbox

[darkrain-nl]

I wanted to try but don't have update.nordpool_update so I'll wait for it to be released...

Weird, it should be there if you installed through HACS?

If you did install manually, just download the master branch and overwrite the files in custom_components\nordpool.

edit: thinking of it, it could possible be an experimental feature of HACS (you can enable that in HACS) edit2: confirmed, it's an experimental feature of HACS: https://hacs.xyz/docs/entities/update_entities

You can enable that in HACS configuration: go to your integrations page, click 'HACS' and click 'configure', it's the last checkbox

I finally got to it and just updated it manually for now, blocking calls gone, so all good from my end.

[phsdv]

blocking calls are gone, but current price is not updating anymore. So not an improvement for me.

[hmmbob]

blocking calls are gone, but current price is not updating anymore. So not an improvement for me.

That might be something local though, it works completely fine here.

[phsdv]

just saw that it was waiting for a reboot. Rebooting now. I am sure I already did a reboot, so not sure what is happening.
[edit]it is updating now[/edit]

[RC-Thoughts]

Is the fix released? I see 0.0.14 from May as latest release? I have the loglines and price is not updating after 2024.6 update today.

[rrooggiieerr]

No, it's not released yet. I think that's something @Hellowlol needs to do.

[hmmbob]

Is the fix released? I see 0.0.14 from May as latest release?

You can update to latest code through the steps in my previous comment.

I have the loglines and price is not updating after 2024.6 update today.

That's unrelated though. The loglines are only warnings, there's nothing broken.

[romlisrl]

Current master version worked for me, warnings gone and prices updates still.
HA 2024.6.4

[Hellowlol]

I have created a beta release as i didnt have time to test all features. Please give it ago

[hiatsu0]

EDIT: It appears my more severe issues may be linked to other recorder issue (and here), however the latest beta also does not work and prevents HA full startup.

For the record, a confirmed case here as well, with the beta not fully resolving the issues:

Upgraded from 2024.5 to 2024.7 and noticed a severe issue where the cpu utilization would jump from normal <10% to around 50% (2 cpu VM) for a few hours and eventually recorder would get clogged and stop recording.
(This gave some error about "items in queue reaching high number and to save memory, the recorder was stopped" but I didn't save that).

After reboots this kind of errors were common:

homeassistant.components.recorder.util
components/recorder/util.py:331
Recorder

The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly

homeassistant.components.recorder.util
components/recorder/util.py:610
Recorder

Ended unfinished session (id=229 from 2024-07-08 19:08:12.034699)

Then I also came across the error in the first post which led me here.

Detected blocking call to open inside the event loop by custom integration 'nordpool'...

Used the update.install service pointed above, and it apprently downloaded the master branch, but after that HA would not fully start anymore until I disabled the nordpool integration. Just got stuck in finalizing startup with this in logs:

homeassistant.components.sensor
runner.py:190
Sensor

Setup of sensor platform nordpool is taking over 10 seconds.

Proceeded to go to HACS to reload the integration, enabled Beta-versions and installed the 0.0.15b0. After this the reboot succeeded (once) but the spot price did not update anymore (notification about "integration not providing this entity anymore"). After second reboot HA got stuck finalizing startup again.

So at least in my case the beta does not seem to function and I think I will need to keep the integration disabled to not mess up the recorder.

[rrooggiieerr]

@hiatsu0, the recorder issue is something different and is because you updated to 2024.7, not because of the Nordpool update. See home-assistant/core#117263

[hiatsu0]

@hiatsu0, the recorder issue is something different and is because you updated to 2024.7, not because of the Nordpool update. See home-assistant/core#117263

Thanks @rrooggiieerr! Was not aware of this issue but indeed looks like that as the problems has occurred at 4:00 last night (though once during the day as well).