Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk' #32

mots · 2024-10-16T17:07:18Z

Should this library ship with the component? I installed using HACS in a docker container.

mots · 2024-10-16T17:09:23Z

I ended up installing it via pip and it seems to work now, but this might still be an issue for other users?

Olen · 2024-10-16T18:32:36Z

It's in the manifest, so it should be installed automatically.

@slaxor505: Have you changed anything in your end?

home-assistant-openplantbook/custom_components/openplantbook/manifest.json

Line 20 in 27a9fec

"openplantbook-sdk==0.4.7"

slaxor505 · 2024-10-17T01:19:59Z

The library is in PyPi. I have not changed anything since the release. It would be interesting to see HASS startup log when it is supposed to attempt and install the library automatically using the manifest. I'm wondering @mots if you have some old Python version because the library minimum is Python 3.9, however since you managed to install it via pip then it might be something else preventing its installation during HASS start up. Would you be able to send a log from HASS?

maxim-mityutko · 2024-10-17T19:30:46Z

I have the same error. HASS is deployed as container - home-assistant:2024.10.2
Upgraded the extension via HACS.

Logger: homeassistant.setup
Source: setup.py:334
First occurred: 13:48:24 (1 occurrences)
Last logged: 13:48:24

Setup failed for custom integration 'openplantbook': Unable to import component: No module named 'openplantbook_sdk'
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component
    component = await integration.async_get_component()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component
    self._component_future.result()
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component
    comp = await self.hass.async_add_import_executor_job(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component
    ComponentProtocol, importlib.import_module(self.pkg_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 995, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/config/custom_components/openplantbook/__init__.py", line 11, in <module>
    from openplantbook_sdk import MissingClientIdOrSecret, OpenPlantBookApi
ModuleNotFoundError: No module named 'openplantbook_sdk'

slaxor505 · 2024-10-17T22:05:15Z

Hi Max. Can you provide HASS startup log please?

…

On Fri, 18 Oct 2024, 06:31 Max, ***@***.***> wrote: I have the same error. HASS is deployed as container - home-assistant:2024.10.2 Upgraded the extension via HACS. Logger: homeassistant.setup Source: setup.py:334 First occurred: 13:48:24 (1 occurrences) Last logged: 13:48:24 Setup failed for custom integration 'openplantbook': Unable to import component: No module named 'openplantbook_sdk' Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/setup.py", line 334, in _async_setup_component component = await integration.async_get_component() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1027, in async_get_component self._component_future.result() File "/usr/src/homeassistant/homeassistant/loader.py", line 1007, in async_get_component comp = await self.hass.async_add_import_executor_job( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run result = self.fn(*self.args, **self.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/loader.py", line 1067, in _get_component ComponentProtocol, importlib.import_module(self.pkg_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/util/loop.py", line 200, in protected_loop_func return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1387, in _gcd_import File "<frozen importlib._bootstrap>", line 1360, in _find_and_load File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 935, in _load_unlocked File "<frozen importlib._bootstrap_external>", line 995, in exec_module File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed File "/config/custom_components/openplantbook/__init__.py", line 11, in <module> from openplantbook_sdk import MissingClientIdOrSecret, OpenPlantBookApi ModuleNotFoundError: No module named 'openplantbook_sdk' — Reply to this email directly, view it on GitHub <#32 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADGJ4FDDUQOUMIV3UMOXN6TZ4AF7ZAVCNFSM6AAAAABQB5ZHQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRQGM3DOMZTHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

maxim-mityutko · 2024-10-18T13:49:23Z

hi @slaxor505 , below are the logs that I get after the container restart. (I've also tried to reinstall the extension for good measure)
logs-from-home-assistant-in-home-assistant-5d86c4b867-gjwqg.log

slaxor505 · 2024-10-19T04:10:29Z

Hi @maxim-mityutko. I'm a bit puzzled here as your startup log is very short. I don't see HASS is setting up components. What log level is set? For example my log much more talkative. Can you increase log level maybe? See attached.

If I remove the library from the python then I will see the following message where HASS attempts to install it. I'm wondering if it is some setting in HASS or not generic HASS container.

2024-10-19 10:23:16.637 INFO (SyncWorker_0) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7

hass.log

slaxor505 · 2024-10-20T00:40:44Z

Hi all,

I was trying to replicate the issue but was unable to do so. I removed and downgraded to OPB 1.2 then I reset Docker container (hass config is persistent). Then I started brand the new container with OPB 1.2. I see that hass installs previous OPB integration library from Olen. Checked that integration works fine. Then upgraded to OPB 1.3, on startup HASS installed opb_sdk on startup with no issues. Then again I checked that integration works fine.

I'm quite puzzled why the issue occurs and need your help to troubleshoot it.

maxim-mityutko · 2024-10-20T19:33:22Z

Thanks for investigating @slaxor505, my logs were indeed set to WARNING. After I switched to INFO, I see:

�[32m2024-10-20 21:19:39.129 INFO (SyncWorker_2) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7�[0m

logs-from-home-assistant-in-home-assistant-5d86c4b867-dslsw.log

maxim-mityutko · 2024-10-20T19:42:55Z

And with DEBUG level (I only copy-pasted the relevant parts)...

�[32m2024-10-20 21:34:30.808 INFO (SyncWorker_7) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7�[0m
�[36m2024-10-20 21:34:30.808 DEBUG (SyncWorker_7) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']�[0m

So I see that pip installs the dependency in /config/deps, these are the contents:

 % ls -hla | grep openplant
drwxr-xr-x  2 root root   4 Oct 16 22:15 openplantbook_sdk
drwxr-xr-x  2 root root   9 Oct 16 22:15 openplantbook_sdk-0.4.7.dist-info

 % ls -hla openplantbook_sdk/
total 22K
drwxr-xr-x  2 root root   4 Oct 16 22:15 .
drwxr-xr-x 33 root root  35 Oct 16 22:15 ..
-rw-r--r--  1 root root 158 Oct 16 22:15 __init__.py
-rw-r--r--  1 root root 18K Oct 16 22:15 sdk.py

Olen · 2024-10-20T19:50:38Z

Didn't HA recently do some "smart" changes to the installation order of modules?

I don't have the docs in front of me here, but I seem to remember there was something with some of the other integrations as well, that needed a minor change to ensure the modules are installed in time.

slaxor505 · 2024-10-21T01:59:23Z

I could not find anything like this in docs. https://developers.home-assistant.io/docs/creating_integration_manifest#requirements

I assume you are talking about other components in "dependencies". But it is not about python libraries but HASS components that the component is dependent on. https://developers.home-assistant.io/docs/creating_integration_manifest#dependencies

I'm wondering if it is specific to architecture. Can @maxim-mityutko let me know if you run it on Raspberry PI? I have not tested this on Arm architecture I can try to do so.

slaxor505 · 2024-10-21T11:10:45Z

Thanks for the details @maxim-mityutko I dag deeper and I see that in my docker container config/deps directory is empty and all the python libs are installed in /usr/local/lib/python3.12/site-packages including openplantbook_sdk.

I have question, do you have any other python libs in "deps" directory too or it is only openplantbook_sdk in there? If it is the only library then does the directory actually visible to python interpreter? E.g. in %PATH% variable.

maxim-mityutko · 2024-10-21T12:28:38Z

It is not the only one:

But pretty much all libraries except for json_timeseries that are installed in this path are also present in /usr/local/lib/python3.12/site-packages

And the /config/deps is not in PATH:

kcm01:/usr/local/lib/python3.12/site-packages# echo $PATH
/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

maxim-mityutko · 2024-10-21T12:39:14Z

Seems like a related thread: home-assistant/core#127966

maxim-mityutko · 2024-10-21T13:34:24Z

I'm wondering if it is specific to architecture. Can @maxim-mityutko let me know if you run it on Raspberry PI? I have not tested this on Arm architecture I can try to do so.

It's running on x86 node

maxim-mityutko · 2024-10-21T14:43:36Z

Don't know where and how the HACS dependencies were installed originally (prior to 2024.10).
The sys.path is updated through the async_mount_local_lib_path method:
https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/bootstrap.py#L668-L676

Which in turn uses async_get_user_site to retrieve paths:
https://github.com/home-assistant/core/blob/06ea3a3014a0c2bdb162e723223e91489d90a48f/homeassistant/util/package.py#L133-L150

The problem however is that --user-site argument in args = [sys.executable, "-m", "site", "--user-site"] won't return the path applied in the PYTHONUSERBASE environment variable, this is facilitated by the --user-base key:
https://docs.python.org/3.12/library/site.html#site.USER_BASE

Hence the path does not end up in PATH.

slaxor505 · 2024-10-22T10:51:11Z

Good catch and it does seem that it is getting a known problem. I'm just curious how it works in my case. All libraries in my docker container have been installed in "site-packages" but not in "deps" which is empty. I also use x86 genuine HASS image. I'm pondering what it depends on and what could be a work around. One obvious is to issue "pip install openplantbook-sdk" within container but not really user friendly one.

maxim-mityutko · 2024-10-22T12:01:14Z

What do you see in the logs, is the --target specified?

�[36m2024-10-20 21:34:30.808 DEBUG (SyncWorker_7) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']�[0m

Installing manually can work as a temporaru workaround, but won't persist after container is recreated. So not really an option in the long run.

Do you by any chance have /.dockerenv in your container filesystem?

slaxor505 · 2024-10-23T09:37:49Z

I reset the container and this is what I got from the startup logs.

2024-10-23 20:28:53.232 INFO (SyncWorker_1) [homeassistant.util.package] Attempting install of openplantbook-sdk==0.4.7 2024-10-23 20:28:53.235 DEBUG (SyncWorker_1) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'openplantbook-sdk==0.4.7', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt']

I don't have --target parameter as you can see.

And I do have .dockerenv in the root

drwxr-xr-x 91 root root 4.0K Oct 23 20:28 .
drwxr-xr-x 91 root root 4.0K Oct 23 20:28 ..
-rwxr-xr-x 1 root root 0 Oct 23 20:28 .dockerenv
-rw-r--r-- 1 root root 76 Oct 19 02:12 OFFICIAL_IMAGE
drwxr-xr-x 2 root root 4.0K Jun 23 01:44 bin
drwxr-xr-x 2 root root 12.0K Jun 22 22:24 command
drwxr-xr-x 11 1024 users 4.0K Oct 23 20:28 config
drwxr-xr-x 5 root root 360 Oct 23 20:28 dev
drwxr-xr-x 41 root root 4.0K Oct 23 20:28 etc
drwxr-xr-x 2 root root 4.0K Jun 19 00:16 home
-rwxr-xr-x 1 root root 1.1K Oct 19 02:12 init
drwxr-xr-x 15 root root 4.0K Jun 23 01:44 lib
drwxr-xr-x 5 root root 4.0K Jun 19 00:16 media
drwxr-xr-x 2 root root 4.0K Jun 19 00:16 mnt
drwxr-xr-x 2 root root 4.0K Jun 19 00:16 opt
drwxr-xr-x 6 root root 4.0K Nov 21 2023 package
dr-xr-xr-x 303 root root 0 Oct 23 20:28 proc
drwx------ 7 root root 4.0K Oct 23 20:36 root
drwxr-xr-x 6 root root 4.0K Oct 23 20:28 run
drwxr-xr-x 2 root root 4.0K Jun 23 01:44 sbin
drwxr-xr-x 2 root root 4.0K Jun 19 00:16 srv
dr-xr-xr-x 12 root root 0 Oct 23 20:28 sys
drwxrwxrwt 2 root root 4.0K Oct 23 20:28 tmp
drwxr-xr-x 57 root root 4.0K Oct 23 20:28 usr
drwxr-xr-x 15 root root 4.0K Jun 23 01:44 var

maxim-mityutko · 2024-10-23T11:07:21Z

Ok, then I have a good idea of what's going on.

The packages are installed with install_package method:
https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/util/package.py#L93

There is a bit of conditional logic there that appends --target to the pip isntallation path:

    if target:
        abs_target = os.path.abspath(target)
        args += ["--target", abs_target]
    elif (
        not is_virtual_env()
        and not (any(var in env for var in _UV_ENV_PYTHON_VARS))
        and (abs_target := site.getusersitepackages())
    ):
        # Pip compatibility
        # Uv has currently no support for --user
        # See https://github.com/astral-sh/uv/issues/2077
        # Using workaround to install to site-packages
        # https://github.com/astral-sh/uv/issues/2077#issuecomment-2150406001
        args += ["--python", sys.executable, "--target", abs_target]

It is called from here with kwargs, one of them is target. And here is how those kwargs are created: https://github.com/home-assistant/core/blob/a301d51fb2a69ce2b187a481d5cfa9b7cb30e453/homeassistant/requirements.py#L88-L97
And the interesting part is:

    if not (config_dir is None or pkg_util.is_virtual_env()) and not is_docker:
        kwargs["target"] = os.path.join(config_dir, "deps")

As you can see target parameter will be used and appended if all the conditions are met.

And the is_docker_env method: https://github.com/home-assistant/core/blob/cdf809926b01af7ae1c7595409be1e4c76ba9467/homeassistant/util/package.py#L30-L32

def is_docker_env() -> bool:
    """Return True if we run in a docker env."""
    return Path("/.dockerenv").exists()

I run my container in Kubernetes, i don't have this file, hence after all the chain of logic is applied I get the /config/deps as target. In your case, the target is not set during the instal_package call, and the elif logic also resolves to false, so the libraries are installed in the default path.

Olen · 2024-10-23T11:33:31Z

Nice debugging.

I think this should be raised with the main HA repo. But why does it not affect other integrations you have installed?

maxim-mityutko · 2024-10-23T13:03:11Z

I don't have a lot of custom integrations that come from HACS, I assume that coincidentally the dependencies for those are available as part of normal HASS distribution.

I agree, the issue is not with this extension. I'll post my finding in the HA core thread.

slaxor505 · 2024-10-30T20:19:56Z

It seems that it was fixed in the latest HASS release. @maxim-mityutko does it work for you now?

maxim-mityutko · 2024-11-04T09:59:13Z

@slaxor505 , nope same error on 2024.10.4

slaxor505 · 2024-11-07T10:35:49Z

The workaround from parallel thread:
home-assistant/core#127966 (comment)

maxim-mityutko · 2024-11-25T00:42:18Z

I've decided to go with dummy .dockerenv file approach. The drawback is that dependencies are not persistent any longer, but I don't have too many 3rd party plugins that require extra deps to be installed. The openplantbook extension is working now, and at first glance nothing has been broken. It's not the ideal approach, but it works.

maxim-mityutko mentioned this issue Oct 23, 2024

Integrations that require a Python module installation fail to add on 2024.10.1 when installed through HACS home-assistant/core#127966

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk' #32

Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk' #32

mots commented Oct 16, 2024

mots commented Oct 16, 2024

Olen commented Oct 16, 2024

slaxor505 commented Oct 17, 2024

maxim-mityutko commented Oct 17, 2024

slaxor505 commented Oct 17, 2024 via email

maxim-mityutko commented Oct 18, 2024

slaxor505 commented Oct 19, 2024 •

edited

Loading

slaxor505 commented Oct 20, 2024

maxim-mityutko commented Oct 20, 2024

maxim-mityutko commented Oct 20, 2024

Olen commented Oct 20, 2024

slaxor505 commented Oct 21, 2024

slaxor505 commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024 •

edited

Loading

slaxor505 commented Oct 22, 2024

maxim-mityutko commented Oct 22, 2024 •

edited

Loading

slaxor505 commented Oct 23, 2024 •

edited

Loading

maxim-mityutko commented Oct 23, 2024 •

edited

Loading

Olen commented Oct 23, 2024

maxim-mityutko commented Oct 23, 2024

slaxor505 commented Oct 30, 2024

maxim-mityutko commented Nov 4, 2024

slaxor505 commented Nov 7, 2024

maxim-mityutko commented Nov 25, 2024

Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk' #32

Error occurred loading flow for integration openplantbook: No module named 'openplantbook_sdk' #32

Comments

mots commented Oct 16, 2024

mots commented Oct 16, 2024

Olen commented Oct 16, 2024

slaxor505 commented Oct 17, 2024

maxim-mityutko commented Oct 17, 2024

slaxor505 commented Oct 17, 2024 via email

maxim-mityutko commented Oct 18, 2024

slaxor505 commented Oct 19, 2024 • edited Loading

slaxor505 commented Oct 20, 2024

maxim-mityutko commented Oct 20, 2024

maxim-mityutko commented Oct 20, 2024

Olen commented Oct 20, 2024

slaxor505 commented Oct 21, 2024

slaxor505 commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024

maxim-mityutko commented Oct 21, 2024 • edited Loading

slaxor505 commented Oct 22, 2024

maxim-mityutko commented Oct 22, 2024 • edited Loading

slaxor505 commented Oct 23, 2024 • edited Loading

maxim-mityutko commented Oct 23, 2024 • edited Loading

Olen commented Oct 23, 2024

maxim-mityutko commented Oct 23, 2024

slaxor505 commented Oct 30, 2024

maxim-mityutko commented Nov 4, 2024

slaxor505 commented Nov 7, 2024

maxim-mityutko commented Nov 25, 2024

slaxor505 commented Oct 19, 2024 •

edited

Loading

maxim-mityutko commented Oct 21, 2024 •

edited

Loading

maxim-mityutko commented Oct 22, 2024 •

edited

Loading

slaxor505 commented Oct 23, 2024 •

edited

Loading

maxim-mityutko commented Oct 23, 2024 •

edited

Loading