-
-
Notifications
You must be signed in to change notification settings - Fork 32.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Integrations that require a Python module installation fail to add on 2024.10.1 when installed through HACS #127966
Comments
This comment was marked as abuse.
This comment was marked as abuse.
That is incorrect, components installed through HACs and others can include that information in their manifest file. And this is evidenced by the information from OP about the logs referencing failing to install those modules. |
This comment was marked as abuse.
This comment was marked as abuse.
You are anything but kind. |
I might have further datapoints for this. First of all, I'm new to home assistant, I started with 2024.09.something. I think this is relevant. I installed the dirigera_platform during the 2024.09 era. Everything works I have since upgraded to 2024.10.2. I'm now having issues with certain extensions
|
@MasseR this is coherent with what I observed as well.
I can confirm symlinking Not sure where to look next to understand where those two diverging paths come from. |
Hi @X-dark :) I see you found this thread too. I was just looking around to report the same, and it looks like you have already done so, thank you! I'm experiencing this also. Packages are getting installed to /config/deps, when HA seems to want to find them at /config/deps/lib/python3.12/site-packages. Subscribing to this thread to see if we get any updates or assistance here. |
Don't know where and how the HACS dependencies were installed originally (prior to 2024.10), currently they are located in The core/homeassistant/bootstrap.py Lines 668 to 676 in a301d51
Which in turn uses core/homeassistant/util/package.py Lines 133 to 150 in 06ea3a3
The problem however is that Hence the path does not end up in PATH. That's my understanding right now how the process works. |
Folks I did a bit more digging. There are 2 questions:
The reason why it's not in the PATH is mentioned in my post above. On the question 1, this logic is influenced by the factors like, is HA is running in docker or venv or some other type of environment. I faced the issue with OpenPlantBook extension, and while it was working perfectly well for some folks, I faced the issue due to the fact that my dependencies were installed in The packages are installed with core/homeassistant/util/package.py Lines 124 to 137 in a301d51
The method is called with a bunch of kwargs, that are generated within the core/homeassistant/scripts/__init__.py Lines 54 to 64 in a301d51
In core/homeassistant/requirements.py Lines 88 to 97 in 487593a
For instance core/homeassistant/util/package.py Lines 30 to 32 in cdf8099
So with that being said, we need to figure what fix is feasible - adding proper catalogs to path or dealing with package installation target. |
Thanks for digging this deep to troubleshoot the error. Regarding your last question: I think we need to understand why the |
I came to this issue, as many users of my integration were facing the same issues you describe with OpenPlantBook. If needed we could ask them which configuration they are using where the issue exists as well. From what I've already seen are many using HAOS. |
Yeah, it would be nice to understand which envs are susceptible to this problem. Somehow I feel that this is related to the transition to |
@FL550 do you know how HAOS manages packages, is it venv? |
As a workaround I set |
Running HA OS, same issue, is there a workaround available? |
@maxim-mityutko raises an interesting point with him running the container in kubernetes. I also ran into the issue and for me the container was running in podman. It seems like the |
Doing some initial digging, to fix this issue for podman and kubernetes at least, the check should be extended to also check for
|
Maybe a fallback environment variable would be nice to have, as I can set it manually in each environment if needed. EDIT: |
@mpreu I agree, I think the environment variable should be used to indicate that HA is running in the containerized environment. More over, this environment variable can be included in the HA dockerfile by default, so you don't even need to set it on your own. |
Just updated core to latest dev, frank_energie and openplantbook integrations now do load, now it's google_home not loading (possibly different issue) maybe it's fixed in dev? |
Well, the only change after the previously mentioned PR is 59e5eb9, so that's the only one that could have fixed it. Essentially just adding a |
Is there a way to finagle it to get it to store deps in /config/deps? |
Yes, setting the env var |
After some investigating, I think I found the reason why it's not loading the dependencies installed into /config/deps
I've verified this by creating a symlink from /config/deps/lib/python3.12/site-packages -> /config/deps In fact, it would be preferable in K8S deployments for it not to detect as docker so it stores all the extra deps in the config dir instead of in /usr/. @edenhaus I believe this is a bug in your fix for uv installing user site-packages? |
tl;dr for everyone else, in the meantime, here's an updated workaround if you want deps installed in /config/deps (You must NOT be using the .dockerenv workaround because that causes HASS to install deps to the python library dir in /usr) home-assistant-0:/config# LIBPATH=$(PYTHONUSERBASE=/config/deps python -m site --user-site); mkdir -p $LIBPATH; rmdir $LIBPATH; ln -s /config/deps $LIBPATH This command should resolve the correct lib path, create the directory structure, and replace the site-packages directory with a symlink to /config/deps, allowing home assistant to load all the dependencies that it installed to the wrong place. If/when it gets fixed in the future, reverting this is basically
|
You shouldn't be using
That is the goal. We persist the layers/changes until an Home Assistant update is performed. Each update, all dependencies are re-installed to ensure compatibility with the current version. |
So, by design, it should be redownloading all extra dependencies on every recreate of the container environment? I understand thats the standard for docker based setups (which HASS OS) is based on, but on K8S environments (which I understand is not officially supported) which some of us use, this does cause a redownload of all deps whenever a pod is restarted, because the container root filesystem is not persistent. It's just that, as it stands, the /config/deps folder provides a convenient way of ensuring it is persisted across Pod restarts for K8S environments. Also, out of curoisity, what is the intended use case for /config/deps? |
Yes, as the environment in the container doesn't have to be compatible with your dependencies anymore. This had lead to tons of issues in the past (e.g. packages present that have been compiled against different versions of ABIs or libraries). |
Unfortunately, that means HA can't properly start without the internet, which for me is very problematic. I guess if deleting /config/deps manually every once in a while is the price to pay not to depend on the internet, I'll take it |
I'm sure you will not like the answer, but maybe the way you run Home Assistant isn't the best way in that case 🤷 |
I mean, you could have an init container, running the same image, that takes a sha256sum of uv pip freeze, and the home assistant version, and wipe that if it changes? Maybe if combined with a venv, hmm. Food for thought. |
@Ulrar Could you give this a look? You can use this script by mounting it over /etc/services.d/home-assistant/run This should, in theory, confine all additional deps to a virtualenv, that is tied to a hash of the current state of packages present in the container image & the machine arch (assumption: this is a fresh env, as is common on kubernetes). This should, in theory, avoid any issues caused by stale dependencies by basically spinning up a new venv on every update, since that changes the hash of uv pip list. If you would like to persist it, just mount a volume into /venv/ |
This is just bad container design. You should never assume that the container root survives a restart. If something needs to persist longer than the runtime of the container, it belongs into a volume/mount. I don't have any issues with the dependencies being re-installed when the version of HA or any of the dependencies itself change, but HA should handle that too, and not simply assume that the dependencies magically get wiped when the version changes. Otherwise anyone using a container runtime that doesn't simply restart old containers will have to build unnessecary workarounds, like building a custom image that needs to be updated for every single plugin or HA update. And simply saying K8s isn't the way to run <x> is something a company distributing some propretary enterprise software would try to pull off, but just seems very inappropriate for a community project of this size. |
That seems to work great. I've made a few tweaks as I'm using the linuxserver image, to run the pod as non root, but it's using the venv and successfully installing the packages as needed, and persisting accross restarts. I've put it under /config/venv to re-use the existing volume. We'll have to see how it goes after an upgrade, but it looks like it'll work as expected, appreciate the solution ! |
I'm running it using user namespaces on my cluster, so it's technically non-root outside the container. Yeah, should probably keep v-1 for easier rollbacks. For me mine is basically at /config/venv, at least for now, I've just added an extra mount to /venv. Technically things like code-server, don't need to be able to access the libraries directory used by home assistant. |
Fair enough. Talos just added user namespaces in the latest release, I haven't enabled it yet but it's a good point, I could maybe switch back to the official image with these. Thanks for the tip, will consider it. In any case I think this solves the issue, it's a bit of extra setup but then again that's what you sign up for by not using HA OS. |
I just removed my work around, and upgraded to 2025.1, no problems so far! For me it's working anyway. Thanks to the team for resolving this! |
Updated to 2025.1 and having issues with custom components. Specifically https://github.com/jdeath/Hubspace-Homeassistant the error is different than the previous error but still does not seem to be working. I am running HomeAssistant via Kubernetes. Will create a new issue after I dig in a bit more but wanted to check if anyone else ran into this issue and has a fix.
I am not able to delete or re-add the integration either. |
@TheAceMan are you running with runAsUser set to a non-root user? If so, this is expected since the system site-packages is owned by root. And the previous container-environment detection bug meant it would install to /config/deps instead (which is writable). Your options are likely to either run as root, enable user namespacing and run as root (https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/), or use a venv like my comment above. |
Thanks @ruifung, I thought the 2025.1 fix addressed all the issues and no modification was required. Is this the long term solution or is there a bug/issue for this remaining functionality. The run file has not changed in 3 years but could break if there are changes to the script being overrode. |
It'll work if you are using the official image and let it run as root, and it has the internet on startup. |
Thanks, and looks like running as root is the only supported option #132336 So this looks like the long term fix without making that change to run as root. |
Or user namespaces. That way, you can be root inside the container but not root outside. Granted, you need, like, the latest 2 versions of K8S, and it's still beta feature flagged. And if you're using talos, the latest version. |
The problem
Hey folks,
I’m in a bit of a pickle trying to add integrations from HACS.
I've recently started with a fresh install of 2024.10.1.
After successfully installing/downloading certain integrations through HACS, I am unable to add them to Home Assistant. When adding it, I get an error dialog with the message:
Config flow could not be loaded: {"message":"Invalid handler specified"}
In the Home Assistant log, it seems to correlate with a message like the following:
2024-10-08 21:40:31.362 INFO (SyncWorker_9) [homeassistant.util.package] Attempting install of meross_iot==0.4.7.3
2024-10-08 21:40:31.362 DEBUG (SyncWorker_9) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'meross_iot==0.4.7.3', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']
2024-10-08 21:40:40.992 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration meross_cloud: No module named 'meross_iot'
That’s specifically when trying to add the "Meross Integration" integration.
It seems integrations that need to install a Python module fail. Integrations that don’t need to install any Python modules, succeed, like themes.
For example, besides the "Meross Integration", the "Tapo Controller" integration also fails to get added, but the Python module in the ERROR log message is different.
Please let me know if you need any more information. As I can reproduce this in a container I can configure and retrieve whatever is needed.
Complete Logs:
home-assistant.tar.gz
Reproduction steps
# Change to docker if needed... podman run -d \ --name homeassistant \ --privileged \ --restart=unless-stopped \ -e TZ=Australia/Brisbane \ -v ./home-assistant:/config \ -v /run/dbus:/run/dbus:ro \ --network=host \ ghcr.io/home-assistant/home-assistant:2024.10.1
Cheers!
What version of Home Assistant Core has the issue?
core-2024.10.1
What was the last working version of Home Assistant Core?
No response
What type of installation are you running?
Home Assistant Container
Integration causing the issue
No response
Link to integration documentation on our website
No response
Diagnostics information
Diagnostics dump
Example YAML snippet
No response
Anything in the logs that might be useful for us?
Additional information
Reported to HACS hacs/integration#4123 but was closed due to not being a HACS issue.
The text was updated successfully, but these errors were encountered: