Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrations that require a Python module installation fail to add on 2024.10.1 when installed through HACS #127966

Closed
joshuar opened this issue Oct 8, 2024 · 79 comments · Fixed by #132404

Comments

@joshuar
Copy link

joshuar commented Oct 8, 2024

The problem

Hey folks,

I’m in a bit of a pickle trying to add integrations from HACS.

I've recently started with a fresh install of 2024.10.1.

After successfully installing/downloading certain integrations through HACS, I am unable to add them to Home Assistant. When adding it, I get an error dialog with the message:

Config flow could not be loaded: {"message":"Invalid handler specified"}

image

In the Home Assistant log, it seems to correlate with a message like the following:

2024-10-08 21:40:31.362 INFO (SyncWorker_9) [homeassistant.util.package] Attempting install of meross_iot==0.4.7.3
2024-10-08 21:40:31.362 DEBUG (SyncWorker_9) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'meross_iot==0.4.7.3', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']
2024-10-08 21:40:40.992 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration meross_cloud: No module named 'meross_iot'

That’s specifically when trying to add the "Meross Integration" integration.

It seems integrations that need to install a Python module fail. Integrations that don’t need to install any Python modules, succeed, like themes.

For example, besides the "Meross Integration", the "Tapo Controller" integration also fails to get added, but the Python module in the ERROR log message is different.

Please let me know if you need any more information. As I can reproduce this in a container I can configure and retrieve whatever is needed.

Complete Logs:

home-assistant.tar.gz

Reproduction steps

  1. Set up a fresh 2024.10.1 container. For e.g.:
# Change to docker if needed...
podman run -d \
            --name homeassistant \
            --privileged \
            --restart=unless-stopped \
            -e TZ=Australia/Brisbane \
            -v ./home-assistant:/config \
            -v /run/dbus:/run/dbus:ro \
            --network=host \
            ghcr.io/home-assistant/home-assistant:2024.10.1
  1. Install HACS as per https://hacs.xyz/docs/use/download/download/#to-download-hacs
  2. Install one of the mentioned integrations through HACS, for example "Meross Integration" or "Tapo Controller".
  3. Try to add said integration to Home Assistant.

Cheers!

What version of Home Assistant Core has the issue?

core-2024.10.1

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant Container

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

Diagnostics dump

{
  "home_assistant": {
    "installation_type": "Unknown",
    "version": "2024.10.1",
    "dev": false,
    "hassio": false,
    "virtualenv": false,
    "python_version": "3.12.4",
    "docker": false,
    "arch": "x86_64",
    "timezone": "Australia/Brisbane",
    "os_name": "Linux",
    "os_version": "6.10.12-200.fc40.x86_64",
    "run_as_root": false
  },
  "custom_components": {
    "hacs": {
      "documentation": "https://hacs.xyz/docs/configuration/start",
      "version": "2.0.1",
      "requirements": [
        "aiogithubapi>=22.10.1"
      ]
    },
    "meross_cloud": {
      "documentation": "https://www.home-assistant.io/components/meross_cloud",
      "version": "1.3.3",
      "requirements": [
        "meross_iot==0.4.7.3"
      ]
    },
    "tapo": {
      "documentation": "https://github.com/petretiandrea/home-assistant-tapo-p100",
      "version": "3.1.2",
      "requirements": [
        "plugp100==5.1.3"
      ]
    }
  },
  "integration_manifest": {
    "domain": "hacs",
    "name": "HACS",
    "after_dependencies": [
      "python_script"
    ],
    "codeowners": [
      "ludeeus"
    ],
    "config_flow": true,
    "dependencies": [
      "http",
      "websocket_api",
      "frontend",
      "persistent_notification",
      "lovelace",
      "repairs"
    ],
    "documentation": "https://hacs.xyz/docs/configuration/start",
    "iot_class": "cloud_polling",
    "issue_tracker": "https://github.com/hacs/integration/issues",
    "requirements": [
      "aiogithubapi>=22.10.1"
    ],
    "version": "2.0.1",
    "is_built_in": false
  },
  "setup_times": {
    "null": {
      "setup": 1.2402997526805848e-05
    },
    "01J9NYGMG1K33YZF6VXF512F8Z": {
      "wait_import_platforms": -0.20091842800320592,
      "wait_base_component": -0.0005247549997875467,
      "config_entry_setup": 0.22828936600126326
    }
  },
  "data": {
    "entry": {
      "created_at": "2024-10-08T11:34:00.449689+00:00",
      "data": {
        "token": "**REDACTED**"
      },
      "discovery_keys": {},
      "disabled_by": null,
      "domain": "hacs",
      "entry_id": "01J9NYGMG1K33YZF6VXF512F8Z",
      "minor_version": 1,
      "modified_at": "2024-10-08T11:34:00.449691+00:00",
      "options": {
        "experimental": true
      },
      "pref_disable_new_entities": false,
      "pref_disable_polling": false,
      "source": "user",
      "title": "",
      "unique_id": null,
      "version": 1
    },
    "hacs": {
      "stage": "running",
      "version": "2.0.1",
      "disabled_reason": null,
      "new": false,
      "startup": false,
      "categories": [
        "theme",
        "integration",
        "plugin",
        "template"
      ],
      "renamed_repositories": {},
      "archived_repositories": [],
      "ignored_repositories": [],
      "lovelace_mode": "storage",
      "configuration": {
        "appdaemon": false,
        "country": "ALL",
        "debug": false,
        "dev": false,
        "python_script": false,
        "release_limit": 5,
        "theme": false
      }
    },
    "custom_repositories": [],
    "repositories": [
      {
        "data": {
          "archived": false,
          "authors": [
            "@ludeeus"
          ],
          "category": "integration",
          "config_flow": false,
          "default_branch": null,
          "description": "HACS gives you a powerful UI to handle downloads of all your custom needs.",
          "domain": "hacs",
          "downloads": 313186,
          "etag_repository": "W/\"305d721bfb9633898fa1781a301dff658c3d0ff2cf40d3e2d349f8932a5791f3\"",
          "etag_releases": null,
          "file_name": "",
          "first_install": false,
          "full_name": "hacs/integration",
          "hide": false,
          "has_issues": true,
          "id": "172733314",
          "installed_commit": null,
          "installed_version": "2.0.1",
          "installed": true,
          "last_commit": null,
          "last_updated": "2024-10-07T14:36:04Z",
          "last_version": "2.0.1",
          "manifest_name": "HACS",
          "new": false,
          "open_issues": 0,
          "prerelease": null,
          "published_tags": [],
          "releases": true,
          "selected_tag": null,
          "show_beta": false,
          "stargazers_count": 5282,
          "topics": [
            "community",
            "package-manager"
          ]
        },
        "integration_manifest": {},
        "repository_manifest": {
          "content_in_root": false,
          "country": [],
          "filename": "hacs.zip",
          "hacs": "0.19.0",
          "hide_default_branch": true,
          "homeassistant": "2024.4.1",
          "manifest": {
            "name": "HACS",
            "zip_release": true,
            "hide_default_branch": true,
            "homeassistant": "2024.4.1",
            "hacs": "0.19.0",
            "filename": "hacs.zip"
          },
          "name": "HACS",
          "persistent_directory": null,
          "render_readme": false,
          "zip_release": true
        },
        "ref": null,
        "paths": {
          "localpath": "/config/custom_components/hacs",
          "local": "/config/custom_components/hacs",
          "remote": "custom_components"
        }
      },
      {
        "data": {
          "archived": false,
          "authors": [
            "@albertogeniola"
          ],
          "category": "integration",
          "config_flow": false,
          "default_branch": null,
          "description": "Custom component that leverages the Meross IoT library to integrate with Homeassistant",
          "domain": "meross_cloud",
          "downloads": 0,
          "etag_repository": "W/\"4e4f614b8b989f05d0f32fed275db661e94c9335cd1bc99b164efcf0986bfcb0\"",
          "etag_releases": null,
          "file_name": "",
          "first_install": false,
          "full_name": "albertogeniola/meross-homeassistant",
          "hide": false,
          "has_issues": true,
          "id": "191831638",
          "installed_commit": "812f589",
          "installed_version": "v1.3.3",
          "installed": true,
          "last_commit": "812f589",
          "last_updated": "2024-09-03T12:09:10Z",
          "last_version": "v1.3.3",
          "manifest_name": "Meross Cloud IoT",
          "new": false,
          "open_issues": 0,
          "prerelease": null,
          "published_tags": [],
          "releases": true,
          "selected_tag": null,
          "show_beta": false,
          "stargazers_count": 721,
          "topics": [
            "meross",
            "meross-homeassistant"
          ]
        },
        "integration_manifest": {},
        "repository_manifest": {
          "content_in_root": false,
          "country": [],
          "filename": null,
          "hacs": "1.6.0",
          "hide_default_branch": false,
          "homeassistant": "2024.1",
          "manifest": {
            "name": "Meross Integration",
            "hacs": "1.6.0",
            "homeassistant": "2024.1"
          },
          "name": "Meross Integration",
          "persistent_directory": null,
          "render_readme": false,
          "zip_release": false
        },
        "ref": null,
        "paths": {
          "localpath": "/config/custom_components/meross_cloud",
          "local": "/config/custom_components/meross_cloud",
          "remote": "custom_components"
        }
      },
      {
        "data": {
          "archived": false,
          "authors": [
            "@petretiandrea"
          ],
          "category": "integration",
          "config_flow": false,
          "default_branch": null,
          "description": "A custom integration to control Tapo devices from home assistant.",
          "domain": "tapo",
          "downloads": 0,
          "etag_repository": "W/\"9c39095aaf36143f5c05a3f85be1d7cf50f372010a4d8002f74b37d8b0e358d4\"",
          "etag_releases": null,
          "file_name": "",
          "first_install": false,
          "full_name": "petretiandrea/home-assistant-tapo-p100",
          "hide": false,
          "has_issues": true,
          "id": "323923603",
          "installed_commit": "c41351b",
          "installed_version": "3.1.4",
          "installed": true,
          "last_commit": "c41351b",
          "last_updated": "2024-10-07T19:02:00Z",
          "last_version": "3.1.4",
          "manifest_name": "TP-Link Tapo",
          "new": false,
          "open_issues": 0,
          "prerelease": null,
          "published_tags": [],
          "releases": true,
          "selected_tag": null,
          "show_beta": false,
          "stargazers_count": 813,
          "topics": [
            "energy",
            "l510",
            "l530",
            "l900",
            "monitoring",
            "p100",
            "p105",
            "p110",
            "smart-plug",
            "tapo",
            "tapo-device",
            "tapo-light-bulb",
            "tp-link",
            "tplink"
          ]
        },
        "integration_manifest": {},
        "repository_manifest": {
          "content_in_root": false,
          "country": [],
          "filename": null,
          "hacs": "1.6.0",
          "hide_default_branch": false,
          "homeassistant": "2024.0.0",
          "manifest": {
            "name": "Tapo Controller",
            "hacs": "1.6.0",
            "render_readme": true,
            "homeassistant": "2024.0.0"
          },
          "name": "Tapo Controller",
          "persistent_directory": null,
          "render_readme": true,
          "zip_release": false
        },
        "ref": null,
        "paths": {
          "localpath": "/config/custom_components/tapo",
          "local": "/config/custom_components/tapo",
          "remote": "custom_components"
        }
      }
    ],
    "rate_limit": {
      "resources": {
        "core": {
          "limit": 5000,
          "used": 0,
          "remaining": 5000,
          "reset": 1728392370
        },
        "search": {
          "limit": 30,
          "used": 0,
          "remaining": 30,
          "reset": 1728388830
        },
        "graphql": {
          "limit": 5000,
          "used": 0,
          "remaining": 5000,
          "reset": 1728392370
        },
        "integration_manifest": {
          "limit": 5000,
          "used": 0,
          "remaining": 5000,
          "reset": 1728392370
        },
        "source_import": {
          "limit": 100,
          "used": 0,
          "remaining": 100,
          "reset": 1728388830
        },
        "code_scanning_upload": {
          "limit": 1000,
          "used": 0,
          "remaining": 1000,
          "reset": 1728392370
        },
        "actions_runner_registration": {
          "limit": 10000,
          "used": 0,
          "remaining": 10000,
          "reset": 1728392370
        },
        "scim": {
          "limit": 15000,
          "used": 0,
          "remaining": 15000,
          "reset": 1728392370
        },
        "dependency_snapshots": {
          "limit": 100,
          "used": 0,
          "remaining": 100,
          "reset": 1728388830
        }
      },
      "rate": {
        "limit": 5000,
        "used": 0,
        "remaining": 5000,
        "reset": 1728392370
      }
    }
  }
}

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-10-08 21:44:19.016 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-08 21:44:19.017 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration meross_cloud which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-08 21:44:19.017 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration tapo which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-08 21:44:19.186 INFO (MainThread) [homeassistant.setup] Setting up system_log
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.setup] Setup of domain logger took 0.03 seconds
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.setup] Setup of domain system_log took 0.03 seconds
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.bootstrap] Setting up frontend: {'frontend'}
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.setup] Setting up http
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.setup] Setting up device_automation
2024-10-08 21:44:19.216 INFO (MainThread) [homeassistant.setup] Setup of domain device_automation took 0.00 seconds
2024-10-08 21:44:19.220 INFO (MainThread) [homeassistant.setup] Setup of domain http took 0.00 seconds
2024-10-08 21:44:19.220 INFO (MainThread) [homeassistant.setup] Setting up auth
2024-10-08 21:44:19.221 INFO (MainThread) [homeassistant.setup] Setup of domain auth took 0.00 seconds
2024-10-08 21:44:19.221 INFO (MainThread) [homeassistant.setup] Setting up config
2024-10-08 21:44:19.223 INFO (MainThread) [homeassistant.setup] Setup of domain config took 0.00 seconds
2024-10-08 21:44:19.223 INFO (MainThread) [homeassistant.setup] Setting up diagnostics
2024-10-08 21:44:19.224 INFO (MainThread) [homeassistant.setup] Setup of domain diagnostics took 0.00 seconds
2024-10-08 21:44:19.224 INFO (MainThread) [homeassistant.setup] Setting up file_upload
2024-10-08 21:44:19.224 INFO (MainThread) [homeassistant.setup] Setup of domain file_upload took 0.00 seconds
2024-10-08 21:44:19.224 INFO (MainThread) [homeassistant.setup] Setting up image_upload
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setup of domain image_upload took 0.00 seconds
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setting up repairs
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setup of domain repairs took 0.00 seconds
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setting up websocket_api
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setup of domain websocket_api took 0.00 seconds
2024-10-08 21:44:19.225 INFO (MainThread) [homeassistant.setup] Setting up api
2024-10-08 21:44:19.226 INFO (MainThread) [homeassistant.setup] Setup of domain api took 0.00 seconds
2024-10-08 21:44:19.226 INFO (MainThread) [homeassistant.setup] Setting up person
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setting up search
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setup of domain search took 0.00 seconds
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setup of domain person took 0.00 seconds
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setting up onboarding
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setup of domain onboarding took 0.00 seconds
2024-10-08 21:44:19.227 INFO (MainThread) [homeassistant.setup] Setting up lovelace
2024-10-08 21:44:19.228 INFO (MainThread) [homeassistant.setup] Setup of domain lovelace took 0.00 seconds
2024-10-08 21:44:19.228 INFO (MainThread) [homeassistant.setup] Setting up frontend
2024-10-08 21:44:19.230 INFO (MainThread) [homeassistant.setup] Setup of domain frontend took 0.00 seconds
2024-10-08 21:44:19.231 INFO (MainThread) [homeassistant.bootstrap] Setting up recorder: {'recorder'}
2024-10-08 21:44:19.231 INFO (MainThread) [homeassistant.setup] Setting up recorder
2024-10-08 21:44:19.233 INFO (MainThread) [homeassistant.components.http] Now listening on port 8123
2024-10-08 21:44:19.233 INFO (MainThread) [homeassistant.setup] Setup of domain http took 0.00 seconds
2024-10-08 21:44:19.273 INFO (MainThread) [homeassistant.setup] Setup of domain recorder took 0.04 seconds
2024-10-08 21:44:19.273 INFO (MainThread) [homeassistant.bootstrap] Setting up stage 1: {'usb', 'websocket_api', 'api', 'network', 'repairs', 'zeroconf', 'http', 'bluetooth', 'auth', 'webhook', 'dhcp', 'ssdp', 'cloud'}
2024-10-08 21:44:19.274 INFO (MainThread) [homeassistant.setup] Setting up network
2024-10-08 21:44:19.276 INFO (MainThread) [homeassistant.setup] Setup of domain network took 0.00 seconds
2024-10-08 21:44:19.277 INFO (MainThread) [homeassistant.setup] Setting up webhook
2024-10-08 21:44:19.277 INFO (MainThread) [homeassistant.setup] Setup of domain webhook took 0.00 seconds
2024-10-08 21:44:19.292 INFO (MainThread) [homeassistant.setup] Setting up usb
2024-10-08 21:44:19.321 INFO (MainThread) [homeassistant.setup] Setup of domain usb took 0.03 seconds
2024-10-08 21:44:19.324 INFO (MainThread) [homeassistant.setup] Setting up zeroconf
2024-10-08 21:44:19.327 INFO (MainThread) [homeassistant.components.zeroconf] Starting Zeroconf broadcast
2024-10-08 21:44:19.327 INFO (MainThread) [homeassistant.setup] Setup of domain zeroconf took 0.00 seconds
2024-10-08 21:44:19.530 INFO (MainThread) [homeassistant.setup] Setting up dhcp
2024-10-08 21:44:19.530 INFO (MainThread) [homeassistant.setup] Setup of domain dhcp took 0.00 seconds
2024-10-08 21:44:19.562 INFO (MainThread) [homeassistant.setup] Setting up ssdp
2024-10-08 21:44:19.563 INFO (MainThread) [homeassistant.setup] Setup of domain ssdp took 0.00 seconds
2024-10-08 21:44:19.719 INFO (MainThread) [homeassistant.setup] Setting up cloud
2024-10-08 21:44:19.721 INFO (MainThread) [homeassistant.setup] Setting up ffmpeg
2024-10-08 21:44:19.721 INFO (MainThread) [homeassistant.setup] Setup of domain ffmpeg took 0.00 seconds
2024-10-08 21:44:19.721 INFO (MainThread) [homeassistant.setup] Setting up tts
2024-10-08 21:44:19.721 INFO (MainThread) [homeassistant.setup] Setup of domain cloud took 0.00 seconds
2024-10-08 21:44:19.723 INFO (MainThread) [homeassistant.setup] Setup of domain tts took 0.00 seconds
2024-10-08 21:44:19.777 INFO (MainThread) [homeassistant.setup] Setting up bluetooth
2024-10-08 21:44:19.792 INFO (MainThread) [homeassistant.setup] Setup of domain bluetooth took 0.01 seconds
2024-10-08 21:44:19.796 INFO (MainThread) [homeassistant.bootstrap] Setting up stage 2: {'map', 'met', 'trace', 'assist_pipeline', 'tag', 'input_button', 'cast', 'logbook', 'default_config', 'diagnostics', 'automation', 'system_health', 'stream', 'wake_word', 'mobile_app', 'stt', 'google_translate', 'person', 'zone', 'application_credentials', 'timer', 'homeassistant_alerts', 'sun', 'file_upload', 'persistent_notification', 'shopping_list', 'ffmpeg', 'input_select', 'blueprint', 'conversation', 'my', 'hardware', 'search', 'device_automation', 'input_text', 'onboarding', 'counter', 'hacs', 'media_source', 'input_number', 'script', 'radio_browser', 'input_datetime', 'tts', 'history', 'intent', 'analytics', 'backup', 'energy', 'schedule', 'input_boolean', 'scene', 'image_upload', 'lovelace', 'config'}
2024-10-08 21:44:19.796 INFO (MainThread) [homeassistant.setup] Setting up wake_word
2024-10-08 21:44:19.797 INFO (MainThread) [homeassistant.setup] Setup of domain wake_word took 0.00 seconds
2024-10-08 21:44:19.797 INFO (MainThread) [homeassistant.setup] Setting up stt
2024-10-08 21:44:19.797 INFO (MainThread) [homeassistant.setup] Setup of domain stt took 0.00 seconds
2024-10-08 21:44:19.797 INFO (MainThread) [homeassistant.setup] Setting up scene
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setup of domain scene took 0.00 seconds
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setting up blueprint
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setup of domain blueprint took 0.00 seconds
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setting up trace
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setup of domain trace took 0.00 seconds
2024-10-08 21:44:19.798 INFO (MainThread) [homeassistant.setup] Setting up input_button
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setup of domain input_button took 0.00 seconds
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setting up logbook
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setup of domain logbook took 0.00 seconds
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setting up history
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setup of domain history took 0.00 seconds
2024-10-08 21:44:19.799 INFO (MainThread) [homeassistant.setup] Setting up media_source
2024-10-08 21:44:19.800 INFO (MainThread) [homeassistant.setup] Setup of domain media_source took 0.00 seconds
2024-10-08 21:44:19.800 INFO (MainThread) [homeassistant.setup] Setting up stream
2024-10-08 21:44:19.800 INFO (MainThread) [homeassistant.setup] Setting up automation
2024-10-08 21:44:19.801 INFO (MainThread) [homeassistant.setup] Setup of domain automation took 0.00 seconds
2024-10-08 21:44:19.801 INFO (MainThread) [homeassistant.setup] Setting up input_number
2024-10-08 21:44:19.802 INFO (MainThread) [homeassistant.setup] Setup of domain input_number took 0.00 seconds
2024-10-08 21:44:19.802 INFO (MainThread) [homeassistant.setup] Setting up system_health
2024-10-08 21:44:19.802 INFO (MainThread) [homeassistant.setup] Setup of domain system_health took 0.00 seconds
2024-10-08 21:44:19.802 INFO (MainThread) [homeassistant.setup] Setting up script
2024-10-08 21:44:19.804 INFO (MainThread) [homeassistant.setup] Setup of domain script took 0.00 seconds
2024-10-08 21:44:19.804 INFO (MainThread) [homeassistant.setup] Setting up zone
2024-10-08 21:44:19.805 INFO (MainThread) [homeassistant.setup] Setup of domain zone took 0.00 seconds
2024-10-08 21:44:19.805 INFO (MainThread) [homeassistant.setup] Setting up timer
2024-10-08 21:44:19.805 INFO (MainThread) [homeassistant.setup] Setup of domain timer took 0.00 seconds
2024-10-08 21:44:19.805 INFO (MainThread) [homeassistant.setup] Setting up input_boolean
2024-10-08 21:44:19.806 INFO (MainThread) [homeassistant.setup] Setup of domain input_boolean took 0.00 seconds
2024-10-08 21:44:19.806 INFO (MainThread) [homeassistant.setup] Setting up input_select
2024-10-08 21:44:19.806 INFO (MainThread) [homeassistant.setup] Setup of domain input_select took 0.00 seconds
2024-10-08 21:44:19.807 INFO (MainThread) [homeassistant.components.scene] Setting up homeassistant.scene
2024-10-08 21:44:19.835 INFO (MainThread) [homeassistant.setup] Setting up intent
2024-10-08 21:44:19.835 INFO (MainThread) [homeassistant.setup] Setup of domain intent took 0.00 seconds
2024-10-08 21:44:19.836 INFO (MainThread) [homeassistant.setup] Setting up conversation
2024-10-08 21:44:19.836 INFO (MainThread) [homeassistant.setup] Setup of domain conversation took 0.00 seconds
2024-10-08 21:44:19.836 INFO (MainThread) [homeassistant.setup] Setting up assist_pipeline
2024-10-08 21:44:19.836 INFO (MainThread) [homeassistant.setup] Setup of domain assist_pipeline took 0.00 seconds
2024-10-08 21:44:19.838 INFO (MainThread) [homeassistant.setup] Setting up map
2024-10-08 21:44:19.838 INFO (MainThread) [homeassistant.setup] Setup of domain map took 0.00 seconds
2024-10-08 21:44:19.839 INFO (MainThread) [homeassistant.setup] Setting up my
2024-10-08 21:44:19.839 INFO (MainThread) [homeassistant.setup] Setup of domain my took 0.00 seconds
2024-10-08 21:44:19.848 INFO (MainThread) [homeassistant.setup] Setting up met
2024-10-08 21:44:19.849 INFO (MainThread) [homeassistant.setup] Setup of domain met took 0.00 seconds
2024-10-08 21:44:19.853 INFO (MainThread) [homeassistant.setup] Setting up hardware
2024-10-08 21:44:19.870 INFO (MainThread) [homeassistant.setup] Setting up tag
2024-10-08 21:44:19.870 INFO (MainThread) [homeassistant.setup] Setup of domain tag took 0.00 seconds
2024-10-08 21:44:19.879 INFO (MainThread) [homeassistant.setup] Setting up input_text
2024-10-08 21:44:19.880 INFO (MainThread) [homeassistant.setup] Setup of domain input_text took 0.00 seconds
2024-10-08 21:44:19.881 INFO (MainThread) [homeassistant.setup] Setting up counter
2024-10-08 21:44:19.881 INFO (MainThread) [homeassistant.setup] Setup of domain counter took 0.00 seconds
2024-10-08 21:44:19.901 INFO (MainThread) [homeassistant.setup] Setting up energy
2024-10-08 21:44:19.901 INFO (MainThread) [homeassistant.setup] Setting up sensor
2024-10-08 21:44:19.901 INFO (MainThread) [homeassistant.setup] Setup of domain sensor took 0.00 seconds
2024-10-08 21:44:19.901 INFO (MainThread) [homeassistant.setup] Setup of domain energy took 0.00 seconds
2024-10-08 21:44:19.902 INFO (MainThread) [homeassistant.setup] Setup of domain hardware took 0.05 seconds
2024-10-08 21:44:19.908 INFO (MainThread) [homeassistant.setup] Setting up homeassistant_alerts
2024-10-08 21:44:19.909 INFO (MainThread) [homeassistant.setup] Setup of domain homeassistant_alerts took 0.00 seconds
2024-10-08 21:44:19.914 INFO (MainThread) [homeassistant.setup] Setup of domain stream took 0.11 seconds
2024-10-08 21:44:19.914 INFO (MainThread) [homeassistant.setup] Setting up sun
2024-10-08 21:44:19.915 INFO (MainThread) [homeassistant.setup] Setup of domain sun took 0.00 seconds
2024-10-08 21:44:19.916 INFO (MainThread) [homeassistant.components.sensor] Setting up sun.sensor
2024-10-08 21:44:19.965 INFO (MainThread) [homeassistant.setup] Setting up hacs
2024-10-08 21:44:19.965 INFO (MainThread) [homeassistant.setup] Setup of domain hacs took 0.00 seconds
2024-10-08 21:44:19.965 INFO (MainThread) [custom_components.hacs] 
-------------------------------------------------------------------
HACS (Home Assistant Community Store)

Version: 2.0.1
This is a custom integration
If you have any issues with this you need to open an issue here:
https://github.com/hacs/integration/issues
-------------------------------------------------------------------

2024-10-08 21:44:19.970 INFO (MainThread) [custom_components.hacs] <HacsData restore> Restore started
2024-10-08 21:44:19.987 INFO (MainThread) [custom_components.hacs] <HacsData restore> Restore done
2024-10-08 21:44:19.988 INFO (MainThread) [custom_components.hacs] Enable category: integration
2024-10-08 21:44:19.988 INFO (MainThread) [custom_components.hacs] Enable category: plugin
2024-10-08 21:44:19.988 INFO (MainThread) [custom_components.hacs] Enable category: template
2024-10-08 21:44:19.988 INFO (MainThread) [custom_components.hacs] Enable category: theme
2024-10-08 21:44:20.022 INFO (MainThread) [homeassistant.setup] Setting up radio_browser
2024-10-08 21:44:20.022 INFO (MainThread) [homeassistant.setup] Setup of domain radio_browser took 0.00 seconds
2024-10-08 21:44:20.023 INFO (MainThread) [homeassistant.setup] Setting up input_datetime
2024-10-08 21:44:20.024 INFO (MainThread) [homeassistant.setup] Setup of domain input_datetime took 0.00 seconds
2024-10-08 21:44:20.025 INFO (MainThread) [homeassistant.setup] Setting up google_translate
2024-10-08 21:44:20.025 INFO (MainThread) [homeassistant.setup] Setup of domain google_translate took 0.00 seconds
2024-10-08 21:44:20.026 INFO (MainThread) [homeassistant.setup] Setting up application_credentials
2024-10-08 21:44:20.026 INFO (MainThread) [homeassistant.setup] Setup of domain application_credentials took 0.00 seconds
2024-10-08 21:44:20.032 INFO (MainThread) [homeassistant.setup] Setting up backup
2024-10-08 21:44:20.032 INFO (MainThread) [homeassistant.setup] Setup of domain backup took 0.00 seconds
2024-10-08 21:44:20.035 INFO (MainThread) [homeassistant.setup] Setting up schedule
2024-10-08 21:44:20.036 INFO (MainThread) [homeassistant.setup] Setup of domain schedule took 0.00 seconds
2024-10-08 21:44:20.037 INFO (MainThread) [homeassistant.setup] Setting up shopping_list
2024-10-08 21:44:20.037 INFO (MainThread) [homeassistant.setup] Setup of domain shopping_list took 0.00 seconds
2024-10-08 21:44:20.097 INFO (MainThread) [homeassistant.setup] Setting up cast
2024-10-08 21:44:20.098 INFO (MainThread) [homeassistant.setup] Setup of domain cast took 0.00 seconds
2024-10-08 21:44:20.180 INFO (MainThread) [homeassistant.setup] Setting up mobile_app
2024-10-08 21:44:20.180 INFO (MainThread) [homeassistant.setup] Setting up notify
2024-10-08 21:44:20.180 INFO (MainThread) [homeassistant.setup] Setup of domain notify took 0.00 seconds
2024-10-08 21:44:20.181 INFO (MainThread) [homeassistant.setup] Setup of domain mobile_app took 0.00 seconds
2024-10-08 21:44:20.181 INFO (MainThread) [homeassistant.components.notify] Setting up notify.mobile_app
2024-10-08 21:44:20.181 INFO (MainThread) [homeassistant.setup] Setting up default_config
2024-10-08 21:44:20.181 INFO (MainThread) [homeassistant.setup] Setup of domain default_config took 0.00 seconds
2024-10-08 21:44:20.184 INFO (MainThread) [homeassistant.components.sensor] Setting up energy.sensor
2024-10-08 21:44:20.186 INFO (MainThread) [homeassistant.setup] Setting up analytics
2024-10-08 21:44:20.186 INFO (MainThread) [homeassistant.setup] Setup of domain analytics took 0.00 seconds
2024-10-08 21:44:20.191 INFO (MainThread) [homeassistant.setup] Setting up switch
2024-10-08 21:44:20.191 INFO (MainThread) [homeassistant.setup] Setup of domain switch took 0.00 seconds
2024-10-08 21:44:20.191 INFO (MainThread) [homeassistant.components.switch] Setting up hacs.switch
2024-10-08 21:44:20.191 INFO (MainThread) [homeassistant.helpers.entity_registry] Registered new switch.hacs entity: switch.tapo_controller_pre_release
2024-10-08 21:44:20.192 INFO (MainThread) [homeassistant.setup] Setting up update
2024-10-08 21:44:20.192 INFO (MainThread) [homeassistant.setup] Setup of domain update took 0.00 seconds
2024-10-08 21:44:20.192 INFO (MainThread) [homeassistant.components.update] Setting up hacs.update
2024-10-08 21:44:20.193 INFO (MainThread) [custom_components.hacs] Stage changed: setup
2024-10-08 21:44:20.193 INFO (MainThread) [custom_components.hacs] Stage changed: waiting
2024-10-08 21:44:20.193 INFO (MainThread) [custom_components.hacs] Setup complete, waiting for Home Assistant before startup tasks starts
2024-10-08 21:44:20.223 INFO (MainThread) [homeassistant.components.tts] Setting up google_translate.tts
2024-10-08 21:44:20.225 INFO (MainThread) [homeassistant.setup] Setting up todo
2024-10-08 21:44:20.227 INFO (MainThread) [homeassistant.setup] Setup of domain todo took 0.00 seconds
2024-10-08 21:44:20.227 INFO (MainThread) [homeassistant.components.todo] Setting up shopping_list.todo
2024-10-08 21:44:20.241 INFO (MainThread) [homeassistant.setup] Setting up media_player
2024-10-08 21:44:20.242 INFO (MainThread) [homeassistant.setup] Setup of domain media_player took 0.00 seconds
2024-10-08 21:44:20.243 INFO (MainThread) [homeassistant.components.media_player] Setting up cast.media_player
2024-10-08 21:44:20.756 INFO (MainThread) [homeassistant.setup] Setting up weather
2024-10-08 21:44:20.757 INFO (MainThread) [homeassistant.setup] Setup of domain weather took 0.00 seconds
2024-10-08 21:44:20.757 INFO (MainThread) [homeassistant.components.weather] Setting up met.weather
2024-10-08 21:44:20.881 INFO (MainThread) [homeassistant.bootstrap] Home Assistant initialized in 1.87s
2024-10-08 21:44:20.882 INFO (MainThread) [homeassistant.core] Starting Home Assistant
2024-10-08 21:44:20.882 INFO (MainThread) [custom_components.hacs] Stage changed: startup
2024-10-08 21:44:20.883 DEBUG (MainThread) [custom_components.hacs] There are 6 scheduled recurring tasks
2024-10-08 21:44:20.883 INFO (MainThread) [custom_components.hacs] Loading removed repositories
2024-10-08 21:44:20.908 INFO (MainThread) [custom_components.hacs] Loading known repositories
2024-10-08 21:44:20.908 DEBUG (MainThread) [custom_components.hacs] Fetching updated content for theme
2024-10-08 21:44:20.909 DEBUG (MainThread) [custom_components.hacs] Fetching updated content for integration
2024-10-08 21:44:20.909 DEBUG (MainThread) [custom_components.hacs] Fetching updated content for plugin
2024-10-08 21:44:20.909 DEBUG (MainThread) [custom_components.hacs] Fetching updated content for template
2024-10-08 21:44:20.993 INFO (MainThread) [custom_components.hacs] Stage changed: running
2024-10-08 21:44:21.002 DEBUG (MainThread) [custom_components.hacs] <QueueManager> The queue is empty
2024-10-08 21:44:21.002 DEBUG (MainThread) [custom_components.hacs] <HACSStore async_save_to_store> Did not store data for 'hacs.critical'. Content did not change
2024-10-08 21:44:21.002 DEBUG (MainThread) [custom_components.hacs] Nothing in the queue
2024-10-08 21:44:21.012 INFO (SyncWorker_4) [homeassistant.loader] Loaded homekit_controller from homeassistant.components.homekit_controller
2024-10-08 21:44:21.012 INFO (SyncWorker_8) [homeassistant.loader] Loaded lifx from homeassistant.components.lifx
2024-10-08 21:44:21.012 INFO (SyncWorker_9) [homeassistant.loader] Loaded kegtron from homeassistant.components.kegtron
2024-10-08 21:44:21.012 INFO (SyncWorker_6) [homeassistant.loader] Loaded androidtv_remote from homeassistant.components.androidtv_remote
2024-10-08 21:44:21.013 INFO (SyncWorker_5) [homeassistant.loader] Loaded synology_dsm from homeassistant.components.synology_dsm
2024-10-08 21:44:21.048 INFO (SyncWorker_1) [homeassistant.loader] Loaded bluetooth_adapters from homeassistant.components.bluetooth_adapters
2024-10-08 21:44:21.060 INFO (SyncWorker_2) [homeassistant.loader] Loaded shelly from homeassistant.components.shelly
2024-10-08 21:44:21.094 INFO (SyncWorker_9) [homeassistant.loader] Loaded esphome from homeassistant.components.esphome
2024-10-08 21:44:21.133 INFO (SyncWorker_6) [homeassistant.loader] Loaded mqtt from homeassistant.components.mqtt
2024-10-08 21:44:21.151 INFO (SyncWorker_3) [homeassistant.loader] Loaded ruuvi_gateway from homeassistant.components.ruuvi_gateway
2024-10-08 21:44:21.441 INFO (MainThread) [homeassistant.setup] Setting up bluetooth_adapters
2024-10-08 21:44:21.445 INFO (SyncWorker_8) [homeassistant.loader] Loaded thread from homeassistant.components.thread
2024-10-08 21:44:25.835 ERROR (Thread-8) [pychromecast.socket_client] [Kitchen Speaker(90661d19-3142-1343-7243-6dae15279938.local.):8009] Failed to connect to service MDNSServiceInfo(name='Google-Nest-Mini-90661d193142134372436dae15279938._googlecast._tcp.local.'), retrying in 5.0s
2024-10-08 21:44:45.267 INFO (SyncWorker_0) [homeassistant.util.package] Attempting install of plugp100==5.1.3
2024-10-08 21:44:45.267 DEBUG (SyncWorker_0) [homeassistant.util.package] Running uv pip command: args=['uv', 'pip', 'install', '--quiet', 'plugp100==5.1.3', '--index-strategy', 'unsafe-first-match', '--upgrade', '--constraint', '/usr/src/homeassistant/homeassistant/package_constraints.txt', '--target', '/config/deps']
2024-10-08 21:44:52.375 ERROR (MainThread) [homeassistant.config_entries] Error occurred loading flow for integration tapo: No module named 'plugp100'
2024-10-08 21:54:20.884 DEBUG (MainThread) [custom_components.hacs] Nothing in the queue
2024-10-08 21:59:30.540 DEBUG (MainThread) [aiogithubapi] 'GitHubRateLimitResourcesModel' is missing key 'audit_log' for <class 'dict'>
2024-10-08 21:59:30.540 DEBUG (MainThread) [aiogithubapi] 'GitHubRateLimitResourcesModel' is missing key 'audit_log_streaming' for <class 'dict'>
2024-10-08 21:59:30.540 DEBUG (MainThread) [aiogithubapi] 'GitHubRateLimitResourcesModel' is missing key 'code_search' for <class 'dict'>
2024-10-08 22:04:20.886 DEBUG (MainThread) [custom_components.hacs] Nothing in the queue

Additional information

Reported to HACS hacs/integration#4123 but was closed due to not being a HACS issue.

@hacscred

This comment was marked as abuse.

@RosemaryOrchard
Copy link

Home Assistant Core won't magically know how to do that for random code from the Internet, which HACS is.

That is incorrect, components installed through HACs and others can include that information in their manifest file. And this is evidenced by the information from OP about the logs referencing failing to install those modules.

@hacscred

This comment was marked as abuse.

@Petro31
Copy link
Contributor

Petro31 commented Oct 12, 2024

I kindly ask you to never speak to me again. Thank you.

You are anything but kind.

@MasseR
Copy link

MasseR commented Oct 16, 2024

I might have further datapoints for this. First of all, I'm new to home assistant, I started with 2024.09.something. I think this is relevant.

I installed the dirigera_platform during the 2024.09 era. Everything works

I have since upgraded to 2024.10.2. I'm now having issues with certain extensions

  • The dirigera_platform got updates that required bumping base libraries. I eventually found out that the dirigera dependency was duplicated on the system. One in /config/deps/dirigera (v1.2.0) and one in /config/deps/libs/python3.12/site-packages/dirigera (v1.1.7). The former one had been updated, the latter one didn't. But the integration tried loading the latter.
  • The dreame-bot integration fails loading entirely due to missing dependencies. The first one to fail is the py-mini-racer. The dependency exists under deps but in /config/deps/py_mini_racer. The custom integration can't see this dependency

@X-dark
Copy link

X-dark commented Oct 16, 2024

@MasseR this is coherent with what I observed as well.

uv installs dependencies in /config/deps as we can see from HA startup logs. But custom integrations seems to look for dependencies elsewhere, hence the issue.

I can confirm symlinking /config/deps/lib/python3.12/site-packages to /config/deps fixes integration that were broken because of that (like mentioned in https://github.com/music-assistant/hass-music-assistant/issues/3031).

Not sure where to look next to understand where those two diverging paths come from.

@danielgoepp
Copy link

Hi @X-dark :) I see you found this thread too. I was just looking around to report the same, and it looks like you have already done so, thank you! I'm experiencing this also. Packages are getting installed to /config/deps, when HA seems to want to find them at /config/deps/lib/python3.12/site-packages. Subscribing to this thread to see if we get any updates or assistance here.

@maxim-mityutko
Copy link

maxim-mityutko commented Oct 21, 2024

Don't know where and how the HACS dependencies were installed originally (prior to 2024.10), currently they are located in /config/deps, which as per my understanding should be available in PATH for the libraries to be discoverable.

The sys.path is updated through the async_mount_local_lib_path method:

async def async_mount_local_lib_path(config_dir: str) -> str:
"""Add local library to Python Path.
This function is a coroutine.
"""
deps_dir = os.path.join(config_dir, "deps")
if (lib_dir := await async_get_user_site(deps_dir)) not in sys.path:
sys.path.insert(0, lib_dir)
return deps_dir

Which in turn uses async_get_user_site to retrieve paths:

async def async_get_user_site(deps_dir: str) -> str:
"""Return user local library path.
This function is a coroutine.
"""
env = os.environ.copy()
env["PYTHONUSERBASE"] = os.path.abspath(deps_dir)
args = [sys.executable, "-m", "site", "--user-site"]
process = await asyncio.create_subprocess_exec(
*args,
stdin=asyncio.subprocess.PIPE,
stdout=asyncio.subprocess.PIPE,
stderr=asyncio.subprocess.DEVNULL,
env=env,
close_fds=False, # required for posix_spawn
)
stdout, _ = await process.communicate()
return stdout.decode().strip()

The problem however is that --user-site argument in args = [sys.executable, "-m", "site", "--user-site"] won't return the path applied in the PYTHONUSERBASE environment variable, this is facilitated by the --user-base key:
https://docs.python.org/3.12/library/site.html#site.USER_BASE

Hence the path does not end up in PATH. That's my understanding right now how the process works.

@maxim-mityutko
Copy link

Folks I did a bit more digging. There are 2 questions:

  1. how the target path where the dependencies are installed is determined
  2. and why this target is not added into PATH.

The reason why it's not in the PATH is mentioned in my post above.

On the question 1, this logic is influenced by the factors like, is HA is running in docker or venv or some other type of environment. I faced the issue with OpenPlantBook extension, and while it was working perfectly well for some folks, I faced the issue due to the fact that my dependencies were installed in /config/deps. While @slaxor505 and I are both running HA in contenerized environment, he uses docker and I use Kubernetes. And this makes a ton of difference, Olen/home-assistant-openplantbook#32

The packages are installed with install_package method, there is a bit of conditional logic there that appends --target to the pip isntallation path:

if target:
abs_target = os.path.abspath(target)
args += ["--target", abs_target]
elif (
not is_virtual_env()
and not (any(var in env for var in _UV_ENV_PYTHON_VARS))
and (abs_target := site.getusersitepackages())
):
# Pip compatibility
# Uv has currently no support for --user
# See https://github.com/astral-sh/uv/issues/2077
# Using workaround to install to site-packages
# https://github.com/astral-sh/uv/issues/2077#issuecomment-2150406001
args += ["--python", sys.executable, "--target", abs_target]

The method is called with a bunch of kwargs, that are generated within the pip_kwargs method:

_pip_kwargs = pip_kwargs(config_dir)
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
for req in getattr(script, "REQUIREMENTS", []):
if is_installed(req):
continue
if not install_package(req, **_pip_kwargs):
print("Aborting script, could not install dependency", req)
return 1

In pip_kwargs there is a conditional logic that adds target parameter and as I mentioned above, it use venv / docker criteria.

def pip_kwargs(config_dir: str | None) -> dict[str, Any]:
"""Return keyword arguments for PIP install."""
is_docker = pkg_util.is_docker_env()
kwargs = {
"constraints": os.path.join(os.path.dirname(__file__), CONSTRAINT_FILE),
"timeout": PIP_TIMEOUT,
}
if not (config_dir is None or pkg_util.is_virtual_env()) and not is_docker:
kwargs["target"] = os.path.join(config_dir, "deps")
return kwargs

For instance is_docker_env method:

def is_docker_env() -> bool:
"""Return True if we run in a docker env."""
return Path("/.dockerenv").exists()

So with that being said, we need to figure what fix is feasible - adding proper catalogs to path or dealing with package installation target.

@FL550
Copy link

FL550 commented Oct 23, 2024

Thanks for digging this deep to troubleshoot the error.

Regarding your last question: I think we need to understand why the is_docker_env check was implemented for choosing the installation target. Just correcting the PATH might introduce problems we are currently not aware of.

@FL550
Copy link

FL550 commented Oct 23, 2024

I came to this issue, as many users of my integration were facing the same issues you describe with OpenPlantBook. If needed we could ask them which configuration they are using where the issue exists as well. From what I've already seen are many using HAOS.

@maxim-mityutko
Copy link

maxim-mityutko commented Oct 23, 2024

Yeah, it would be nice to understand which envs are susceptible to this problem.

Somehow I feel that this is related to the transition to uv and this PR: #125808

@maxim-mityutko
Copy link

@FL550 do you know how HAOS manages packages, is it venv?

@CoolDuke
Copy link

As a workaround I set PYTHONPATH to /config/deps in the container's environment so HASS can find and load the custom dependencies.

@tbrasser
Copy link

Running HA OS, same issue, is there a workaround available?

@mots
Copy link

mots commented Oct 26, 2024

@maxim-mityutko raises an interesting point with him running the container in kubernetes. I also ran into the issue and for me the container was running in podman.

It seems like the is_docker_env check should be updated so it works for other container environments.

@mots
Copy link

mots commented Oct 26, 2024

Doing some initial digging, to fix this issue for podman and kubernetes at least, the check should be extended to also check for

/run/containerenv (podman) and
/var/run/secrets/kubernetes.io (kubernetes)

@mpreu
Copy link

mpreu commented Oct 26, 2024

Doing some initial digging, to fix this issue for podman and kubernetes at least, the check should be extended to also check for

/run/containerenv (podman) and /var/run/secrets/kubernetes.io (kubernetes)

/var/run/secrets/kubernetes.io/... is not a fail-safe test for a Kubernetes environment. Using automountServiceAccountToken: false is considered a good practice when an application does not need the k8s API access. In this case the complete directory structure is not populated. IMHO it would be good to not "enforce" the automatic mount.

[var]/run/.containerenv is at least available for cri-o (cri-o/cri-o#5461), but maybe not in other supported runtimes like containerd.

Maybe a fallback environment variable would be nice to have, as I can set it manually in each environment if needed.

EDIT:
Currently, the more viable test for a Kubernetes environment is using the default KUBERNETES_* environment variables (e.g. KUBERNETES_PORT, KUBERNETES_SERVICE_PORT, KUBERNETES_SERVICE_HOST). These injections cannot be disabled easily.

@maxim-mityutko
Copy link

@mpreu I agree, I think the environment variable should be used to indicate that HA is running in the containerized environment. More over, this environment variable can be included in the HA dockerfile by default, so you don't even need to set it on your own.

@tbrasser
Copy link

Just updated core to latest dev, frank_energie and openplantbook integrations now do load, now it's google_home not loading (possibly different issue)

maybe it's fixed in dev?

@towo
Copy link

towo commented Oct 26, 2024

Well, the only change after the previously mentioned PR is 59e5eb9, so that's the only one that could have fixed it. Essentially just adding a -m in there.

@ruifung
Copy link

ruifung commented Jan 1, 2025

Is there a way to finagle it to get it to store deps in /config/deps?
Because having it require redownloading dependencies on every pod recreate is problematic for something that should run offline.

@Ulrar
Copy link

Ulrar commented Jan 1, 2025

Is there a way to finagle it to get it to store deps in /config/deps? Because having it require redownloading dependencies on every pod recreate is problematic for something that should run offline.

Yes, setting the env var PYTHONPATH=/config/deps does that, but you have to fiddle with it sometimes, or at least 2024.12 required me to delete it manually, restart, install a dependency manually in it and restart again, from memory

@ruifung
Copy link

ruifung commented Jan 1, 2025

After some investigating, I think I found the reason why it's not loading the dependencies installed into /config/deps

  1. HASS tries to add /config/deps into sys.path using async_mount_local_lib_path.

    await async_mount_local_lib_path(runtime_config.config_dir)

  2. But it uses async_get_user_site

    if (lib_dir := await async_get_user_site(deps_dir)) not in sys.path:

  3. Which resolves the actual user site-path using python3 -m site --user-site with USERBASEPATH set to /config/deps. This results in the path /config/deps/lib/python3.13/site-packages being injected into the sys.path.

    args = [sys.executable, "-m", "site", "--user-site"]

  4. The changes introduced in d9812f0 to emulate pip's --user support with uv does not take into account the full path for the user-site, and thus installs all the deps directly into /config/deps instead of the expected /config/deps/lib/python3.13/site-packages

    args += ["--target", abs_target]

  5. This results in a mismatch between what HASS expects it to be (the path injected into sys.path), and what it actually installs the additional dependencies into.

I've verified this by creating a symlink from /config/deps/lib/python3.12/site-packages -> /config/deps
And it loads the dependencies properly.
From what I've found, either fixing the UV --target to use the proper site-packages path as returned from the site module should allow /config/deps to be loaded properly when run (even if it's not detected as docker.)

In fact, it would be preferable in K8S deployments for it not to detect as docker so it stores all the extra deps in the config dir instead of in /usr/.

@edenhaus I believe this is a bug in your fix for uv installing user site-packages?

@ruifung
Copy link

ruifung commented Jan 1, 2025

tl;dr for everyone else, in the meantime, here's an updated workaround if you want deps installed in /config/deps (You must NOT be using the .dockerenv workaround because that causes HASS to install deps to the python library dir in /usr)

home-assistant-0:/config# LIBPATH=$(PYTHONUSERBASE=/config/deps python -m site --user-site); mkdir -p $LIBPATH; rmdir $LIBPATH; ln -s /config/deps $LIBPATH

This command should resolve the correct lib path, create the directory structure, and replace the site-packages directory with a symlink to /config/deps, allowing home assistant to load all the dependencies that it installed to the wrong place.

If/when it gets fixed in the future, reverting this is basically

home-assistant-0:/config# rm -r /config/deps/lib

@frenck
Copy link
Member

frenck commented Jan 2, 2025

tl;dr for everyone else, in the meantime, here's an updated workaround if you want deps installed in /config/deps (You must NOT be using the .dockerenv workaround because that causes HASS to install deps to the python library dir in /usr)

You shouldn't be using /config/deps; this is by design.

(You must NOT be using the .dockerenv workaround because that causes HASS to install deps to the python library dir in /usr)

That is the goal. We persist the layers/changes until an Home Assistant update is performed. Each update, all dependencies are re-installed to ensure compatibility with the current version.

@ruifung
Copy link

ruifung commented Jan 2, 2025

So, by design, it should be redownloading all extra dependencies on every recreate of the container environment?

I understand thats the standard for docker based setups (which HASS OS) is based on, but on K8S environments (which I understand is not officially supported) which some of us use, this does cause a redownload of all deps whenever a pod is restarted, because the container root filesystem is not persistent.

It's just that, as it stands, the /config/deps folder provides a convenient way of ensuring it is persisted across Pod restarts for K8S environments.

Also, out of curoisity, what is the intended use case for /config/deps?

@frenck
Copy link
Member

frenck commented Jan 2, 2025

So, by design, it should be redownloading all extra dependencies on every recreate of the container environment?

Yes, as the environment in the container doesn't have to be compatible with your dependencies anymore. This had lead to tons of issues in the past (e.g. packages present that have been compiled against different versions of ABIs or libraries).

@Ulrar
Copy link

Ulrar commented Jan 2, 2025

So, by design, it should be redownloading all extra dependencies on every recreate of the container environment?

Yes, as the environment in the container doesn't have to be compatible with your dependencies anymore. This had lead to tons of issues in the past (e.g. packages present that have been compiled against different versions of ABIs or libraries).

Unfortunately, that means HA can't properly start without the internet, which for me is very problematic. I guess if deleting /config/deps manually every once in a while is the price to pay not to depend on the internet, I'll take it

@frenck
Copy link
Member

frenck commented Jan 2, 2025

I'm sure you will not like the answer, but maybe the way you run Home Assistant isn't the best way in that case 🤷

@ruifung
Copy link

ruifung commented Jan 2, 2025

I mean, you could have an init container, running the same image, that takes a sha256sum of uv pip freeze, and the home assistant version, and wipe that if it changes?

Maybe if combined with a venv, hmm. Food for thought.

@ruifung
Copy link

ruifung commented Jan 2, 2025

@Ulrar Could you give this a look?
https://github.com/ruifung/rfhome-infrastructure/blob/master/deployments/pathweb/workloads/home-network/home-assistant/venv-run

You can use this script by mounting it over /etc/services.d/home-assistant/run

This should, in theory, confine all additional deps to a virtualenv, that is tied to a hash of the current state of packages present in the container image & the machine arch (assumption: this is a fresh env, as is common on kubernetes). This should, in theory, avoid any issues caused by stale dependencies by basically spinning up a new venv on every update, since that changes the hash of uv pip list.

If you would like to persist it, just mount a volume into /venv/

@DerEnderKeks
Copy link
Contributor

(You must NOT be using the .dockerenv workaround because that causes HASS to install deps to the python library dir in /usr)

That is the goal. We persist the layers/changes until an Home Assistant update is performed. Each update, all dependencies are re-installed to ensure compatibility with the current version.

This is just bad container design. You should never assume that the container root survives a restart. If something needs to persist longer than the runtime of the container, it belongs into a volume/mount.

I don't have any issues with the dependencies being re-installed when the version of HA or any of the dependencies itself change, but HA should handle that too, and not simply assume that the dependencies magically get wiped when the version changes.
This could be as simple as a file containing the HA version in the persistent dependencies directory, used by HA to wipe the directory if the version doesn't match the running one. This would result in the intended behaviour, without depending on the container root surving a restart.

Otherwise anyone using a container runtime that doesn't simply restart old containers will have to build unnessecary workarounds, like building a custom image that needs to be updated for every single plugin or HA update. And simply saying K8s isn't the way to run <x> is something a company distributing some propretary enterprise software would try to pull off, but just seems very inappropriate for a community project of this size.

@Ulrar
Copy link

Ulrar commented Jan 2, 2025

@ruifung

#127966 (comment)

That seems to work great. I've made a few tweaks as I'm using the linuxserver image, to run the pod as non root, but it's using the venv and successfully installing the packages as needed, and persisting accross restarts. I've put it under /config/venv to re-use the existing volume.

We'll have to see how it goes after an upgrade, but it looks like it'll work as expected, appreciate the solution !
I'll also add a line to delete older venvs after an upgrade, to avoid it just taking more and more space. Maybe keep v-1 to be safe.

@ruifung
Copy link

ruifung commented Jan 2, 2025

@ruifung

#127966 (comment)

That seems to work great. I've made a few tweaks as I'm using the linuxserver image, to run the pod as non root, but it's using the venv and successfully installing the packages as needed, and persisting accross restarts. I've put it under /config/venv to re-use the existing volume.

We'll have to see how it goes after an upgrade, but it looks like it'll work as expected, appreciate the solution ! I'll also add a line to delete older venvs after an upgrade, to avoid it just taking more and more space. Maybe keep v-1 to be safe.

I'm running it using user namespaces on my cluster, so it's technically non-root outside the container.

Yeah, should probably keep v-1 for easier rollbacks.

For me mine is basically at /config/venv, at least for now, I've just added an extra mount to /venv. Technically things like code-server, don't need to be able to access the libraries directory used by home assistant.

@Ulrar
Copy link

Ulrar commented Jan 2, 2025

I'm running it using user namespaces on my cluster, so it's technically non-root outside the container.

Yeah, should probably keep v-1 for easier rollbacks.

For me mine is basically at /config/venv, at least for now, I've just added an extra mount to /venv. Technically things like code-server, don't need to be able to access the libraries directory used by home assistant.

Fair enough. Talos just added user namespaces in the latest release, I haven't enabled it yet but it's a good point, I could maybe switch back to the official image with these. Thanks for the tip, will consider it.

In any case I think this solves the issue, it's a bit of extra setup but then again that's what you sign up for by not using HA OS.

@danielgoepp
Copy link

I just removed my work around, and upgraded to 2025.1, no problems so far! For me it's working anyway. Thanks to the team for resolving this!

@TheAceMan
Copy link

Updated to 2025.1 and having issues with custom components. Specifically https://github.com/jdeath/Hubspace-Homeassistant the error is different than the previous error but still does not seem to be working. I am running HomeAssistant via Kubernetes. Will create a new issue after I dig in a bit more but wanted to check if anyone else ran into this issue and has a fix.

2025-01-04 00:49:07.931 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package hubspace-async==0.4.1: error: Failed to install: hubspace_async-0.4.1-py3-none-any.whl (hubspace-async==0.4.1)
  Caused by: failed to create directory `/usr/local/lib/python3.13/site-packages/hubspace_async`: Permission denied (os error 13)
2025-01-04 00:49:10.574 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package hubspace-async==0.4.1: error: Failed to install: hubspace_async-0.4.1-py3-none-any.whl (hubspace-async==0.4.1)
  Caused by: failed to create directory `/usr/local/lib/python3.13/site-packages/hubspace_async`: Permission denied (os error 13)
2025-01-04 00:49:13.215 ERROR (SyncWorker_5) [homeassistant.util.package] Unable to install package hubspace-async==0.4.1: error: Failed to install: hubspace_async-0.4.1-py3-none-any.whl (hubspace-async==0.4.1)
  Caused by: failed to create directory `/usr/local/lib/python3.13/site-packages/hubspace_async`: Permission denied (os error 13)
2025-01-04 00:49:13.216 ERROR (MainThread) [homeassistant.setup] Setup failed for custom integration 'hubspace': Requirements for hubspace not found: ['hubspace-async==0.4.1'].

I am not able to delete or re-add the integration either.

@ruifung
Copy link

ruifung commented Jan 5, 2025

@TheAceMan are you running with runAsUser set to a non-root user? If so, this is expected since the system site-packages is owned by root. And the previous container-environment detection bug meant it would install to /config/deps instead (which is writable).

Your options are likely to either run as root, enable user namespacing and run as root (https://kubernetes.io/docs/concepts/workloads/pods/user-namespaces/), or use a venv like my comment above.
#127966 (comment)

@TheAceMan
Copy link

Thanks @ruifung, I thought the 2025.1 fix addressed all the issues and no modification was required.

Is this the long term solution or is there a bug/issue for this remaining functionality. The run file has not changed in 3 years but could break if there are changes to the script being overrode.

@Ulrar
Copy link

Ulrar commented Jan 6, 2025

Thanks @ruifung, I thought the 2025.1 fix addressed all the issues and no modification was required.

Is this the long term solution or is there a bug/issue for this remaining functionality. The run file has not changed in 3 years but could break if there are changes to the script being overrode.

It'll work if you are using the official image and let it run as root, and it has the internet on startup.
If you run as non root, or do not have the internet during each container / pod startup, the dependencies will fail to install if you don't do extra steps (such as use the run script linked above)

@TheAceMan
Copy link

Thanks, and looks like running as root is the only supported option #132336

So this looks like the long term fix without making that change to run as root.

@ruifung
Copy link

ruifung commented Jan 6, 2025

Thanks, and looks like running as root is the only supported option #132336

So this looks like the long term fix without making that change to run as root.

Or user namespaces. That way, you can be root inside the container but not root outside. Granted, you need, like, the latest 2 versions of K8S, and it's still beta feature flagged. And if you're using talos, the latest version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.