Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BIG problem after upgrade HA pyton 3.10 #6117

Closed
maury77 opened this issue Jan 29, 2023 · 21 comments
Closed

BIG problem after upgrade HA pyton 3.10 #6117

maury77 opened this issue Jan 29, 2023 · 21 comments
Labels
Enhancement 💨 Solution available 🥂 Definite solution has been done
Milestone
v8.15

Comments

@maury77
Copy link

maury77 commented Jan 29, 2023 •
edited by MichaIng
Loading

DietPi version |

G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=13
G_DIETPI_VERSION_RC=2
G_GITBRANCH='master'
G_GITOWNER='MichaIng'
G_LIVE_PATCH_STATUS[0]='applied'
G_LIVE_PATCH_STATUS[1]='not applicable'

Distro version | bullseye 1

Kernel version | uname -a
SBC model | root@sdomotica:~# uname -a
Linux sdomotica 5.15.84-v7+ #1613 SMP Thu Jan 5 11:59:48 GMT 2023 armv7l GNU/Linux

SDcard used | (EG: SanDisk ultra 32GB)
Home Assistant system log | journalctl -u home-assistant

After reinstall HA via dietpi I have big issue

feb 08 12:11:53 sdomotica Home Assistant[9286]: sqlalchemy.exc.ArgumentError: Type annotation for "Statistics.metadata_id" can't be correctly interpreted for Annotated Declarative Table form.  ORM annotations should normally make use of the ``Mapped[]`` generic type, or other ORM-compatible generic type, as a container for the actual type, which indicates the intent that the attribute is mapped. Class variables that are not intended to be mapped by the ORM should use ClassVar[].  To allow Annotated Declarative to disregard legacy annotations which don't use Mapped[] to pass, set "__allow_unmapped__ = True" on the class or a superclass this class. (Background on this error at: https://sqlalche.me/e/20/zlpr)
feb 08 12:11:53 sdomotica Home Assistant[9286]: 2023-02-08 12:11:53.888 ERROR (MainThread) [homeassistant.setup] Setup failed for recorder: Unable to import component: Exception importing homeassistant.components.recorder

The HA don't start gui adn service the ripetitive error is
@maury77 maury77 changed the title BIG problema after upgrade HA 3.10 BIG problema after upgrade HA pyton 3.10 Jan 29, 2023
@Joulinar
Copy link
Collaborator

any reason not filling the bug report template?

Required

  • DietPi version | cat /boot/dietpi/.version
  • Distro version | echo $G_DISTRO_NAME $G_RASPBIAN
  • Kernel version | uname -a
  • SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
  • Power supply used | (EG: 5V 1A RAVpower)
  • SDcard used | (EG: SanDisk ultra)
  • Home Assistant system log | journalctl -u home-assistant

@maury77
Copy link
Author

maury77 commented Feb 8, 2023

any Idea I rollback is ok ! now I try upgrade new and the same errore

HELP

@maury77 maury77 changed the title BIG problema after upgrade HA pyton 3.10 BIG problem after upgrade HA pyton 3.10 Feb 8, 2023
@maury77
Copy link
Author

maury77 commented Feb 8, 2023 via email

@MichaIng
Copy link
Owner

MichaIng commented Feb 8, 2023

You did already try to restart the service and wait until all needed dependencies have been installed, watchinghtop?

systemctl restart home-assistant

Otherwise, please try to that again after (re)moving the deps directory:

mv /mnt/dietpi_userdata/homeassistant/deps /mnt/dietpi_userdata/homeassistant/deps.bak
systemctl restart home-assistant
htop

@maury77
Copy link
Author

maury77 commented Feb 8, 2023

I found the problem

The installation with dietpi-software reinstall 157 not install sqlalchemye

I have install the last version (2.02( that is not compatible with stack , I try install 1.46 and the problem is resolved

@MichaIng
Copy link
Owner

MichaIng commented Feb 8, 2023

Ah, then I wonder whether the dependency was missed to be declared in one of the integration you use. Usually HA installs them automatically on startup or when accessing features the first time. dietpi-software installs the modules needed by HA core declared via requirements.txt, for everything else integrations are expected to declare their dependencies internally, so that HA installs them on demand to /mnt/dietpi_userdata/homeassistant/deps.

But great that you found the solution 👍.

@Joulinar
Copy link
Collaborator

Joulinar commented Feb 8, 2023 •
edited
Loading

I guess one of the installed python packages is not compatible with SQLAlchemy v2. Probably some wheels would need to be updated by the individual maintainer https://docs.sqlalchemy.org/en/20/errors.html#error-zlpr

SQLAlchemy v2 has been released 2 weeks back. If not specified, it will be pulled automatically as latest version available https://www.piwheels.org/project/sqlalchemy/

@MichaIng
Copy link
Owner

MichaIng commented Feb 8, 2023

Ah this can be the cause as well of course. I understood it so that SQLAlchemy was not installed OOTB but manually, in which case v2 is pulled. Since the integrations are no Python modules (are they?) and their dependencies are not installed into the system's or pyenv Python environment, but into this dedicated /mnt/dietpi_userdata/homeassistant/deps directory, pip cannot see or warn about conflicts, when you manually install modules.

@maury77
Copy link
Author

maury77 commented Feb 10, 2023

the integrations should all be python modules, here it was the native recorder module

After upgrade there was no SQLAlchemy and I had installed it manually

@MichaIng
Copy link
Owner

the integrations should all be python modules, here it was the native recorder module

That is right, but those are also not installed into the regular Python environment, so pip does not see them. Actually, probably one can make pip have a look into /mnt/dietpi_userdata/homeassistant/deps as well, if that is just some kind of venv with common Python directory structure 🤔.

@maury77
Copy link
Author

maury77 commented Feb 16, 2023 •
edited by MichaIng
Loading 

root@sdomotica:/mnt/dietpi_userdata/homeassistant/deps# ls -al
totale 52
drwxrwxr-x  8 homeassistant homeassistant 4096 27 dic 16.21 .
drwxrwxr-x 15 homeassistant homeassistant 4096 14 feb 16.15 ..
drwxr-xr-x  3 homeassistant homeassistant 4096  8 feb 13.04 bin
drwxr-xr-x  2 homeassistant homeassistant 4096  8 feb 11.53 cache
-rw-r--r--  1 homeassistant homeassistant   15  6 mar  2021 ciso-requirements.txt
drwxr-xr-x  2 homeassistant homeassistant 4096 11 ott 23.22 doc
-rw-r--r--  1 homeassistant homeassistant   21  6 mar  2021 extra-requirements.txt
drwxr-xr-x  3 homeassistant homeassistant 4096  9 mag  2021 include
drwxr-xr-x  5 homeassistant homeassistant 4096  8 feb 11.45 lib
-rw-r--r--  1 homeassistant homeassistant 1065 27 dic 16.21 LICENSE
-rw-r--r--  1 homeassistant homeassistant  121  6 mar  2021 requirements.txt
drwxr-xr-x  3 homeassistant homeassistant 4096 10 feb  2020 share
-rw-r--r--  1 homeassistant homeassistant  114  6 mar  2021 test-requirements.txt

root@sdomotica:/mnt/dietpi_userdata/homeassistant/deps# more requirements.txt 
rx >= 3.0.1
certifi >= 14.05.14
six >= 1.10
python_dateutil >= 2.5.3
setuptools >= 21.0.0
urllib3 >= 1.15.1
pytz>=2019.1

root@sdomotica:/mnt/dietpi_userdata/homeassistant/deps# more extra-requirements.txt 
pandas>=0.25.3
numpy

MichaIng added a commit that referenced this issue Feb 18, 2023
@MichaIng
v8.15
a677610 
- DietPi-Software | Home Assistant: Our Home Assistant integration used two Python environments: A pyenv is used to install the "homeassistant" core module and its direct dependencies. Home Assistant itself installs its frontend with dependencies on startup, as well as dependencies for integrations on demand, internally into another Python environment. On Home Assistant reinstalls, pyenv is removed but the internal dependencies were left untouched. This caused issues when those dependency builds were incompatible with the new pyenv Python version, or conflicting with other pyenv module upgrades. Both environments are now merged and reset on every reinstall, which solves potential conflicts but means that the first Home Assistant startup after a reinstall takes longer. Many thanks to @maury77 for reporting a related issue: #6117
@MichaIng MichaIng added this to the v8.15 milestone Feb 18, 2023
@MichaIng MichaIng added Enhancement 💨 Solution available 🥂 Definite solution has been done labels Feb 18, 2023
@MichaIng
Copy link
Owner

Did you create these requirements.txt manually or is this some old leftover? Based on timestamps looks like the letter. So that is just a common Python environment and can be merged with pyenv's and then reset the same way on reinstalls. This assures that there are no version mismatches but means that all internal dependencies need to be reinstalled, in case recompiled as well.

I tested back and forth, and HA is actually quite robust working with merged pyenv and deps environments. Even if you cause conflicts by upgrading everything to versions not supported by the integrations, HA automatically downgrades affected modules, reinstalls missing ones etc.

Hence: a677610

Little bonus is that one can upgrade or cleanup the internal deps as well, within the borders or what the integrations support. One however cannot break it, as HA automatically assures that all dependencies are met, as mentioned 🙂.

@maury77
Copy link
Author

maury77 commented Feb 21, 2023

I have not created them manually, I think they are old things after several upgrades

@maury77
Copy link
Author

maury77 commented Mar 4, 2023

Sorry , I try to upgrade tu minor version 2023.3.1 but I have new crash with sqlaalchemy and HA don't start

with many error as type Setup failed for recorder: Unable to import component: cannot import name 'ColumnElement' from 'sqlalchemy'

@MichaIng
Copy link
Owner

MichaIng commented Mar 4, 2023 •
edited
Loading

How did you try to upgrade HA, via pip3 in the pyenv, or via reinstall?

Unexpected if the Python version itself has not changed. However, for now, before you reinstall it:

rm -R /mnt/dietpi_userdata/homeassistant/deps

This will be done automatically on reinstalls after next DietPi release.

Ah, you can do this also without a reinstall, and just restart the service afterwards, to solve the issue now:

rm -R /mnt/dietpi_userdata/homeassistant/deps
systemctl restart home-assistent

HA installs all needed dependencies on demand then.

@maury77
Copy link
Author

maury77 commented Mar 4, 2023

I use this script

exec sudo -u homeassistant dash -c '. /home/homeassistant/pyenv-activate.sh; exec pip3 install --no-cache-dir -U homeassistant'

This was referenced Mar 4, 2023
Merged
Closed
@MichaIng
Copy link
Owner

MichaIng commented Mar 4, 2023

A little simpler 🙂:

/home/homeassistant/homeassistant-update.sh

It does the sudo stuff automatically.

But indeed strange how it could have caused that issue then. If any other sqlalchemy version was needed, if would have pulled it as dependency, and as well on service start, this should be part of the automatic deps installs.

@maury77
Copy link
Author

maury77 commented Mar 6, 2023

the problem is the same of the initial issue.

Also upgrade with "/home/homeassistant/homeassistant-update.sh" doesn't install right version of SQLAlchemy

In the 2023.3.1 SQLAlchemy 2.0.2 is necessary and I had the 1.46

I install manualy SQLAlchemy 2.0.2 ando now it is ok

@MichaIng
Copy link
Owner

MichaIng commented Mar 6, 2023

Then there is a dependency conflict within Home Assistent, respectively between Home Assistant and one of your integrations. An older SQLAlchemy wouldn't be installed of not either HA or an integration declares to be not compatible with the recent one.

Do you find SQLAlchemy within /mnt/dietpi_userdata/homeassistant/deps or only with

sudo -u homeassistant dash -c '. /home/homeassistant/pyenv-activate.sh; exec pip3 freeze'

?

@MichaIng MichaIng mentioned this issue Mar 11, 2023
Merged
@cazzoo
Copy link

cazzoo commented Feb 28, 2024 •
edited
Loading

I notice some issues with the latest update as well about webrtc-noise-gain missing. I've been trying to add the package directly but that did complain for missing gcc (expected AFAIK).

Collecting webrtc-noise-gain
  Downloading webrtc_noise_gain-1.2.3.tar.gz (3.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.1/3.1 MB 413.1 kB/s eta 0:00:00

  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: webrtc-noise-gain
  Building wheel for webrtc-noise-gain (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for webrtc-noise-gain (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [77 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-aarch64-cpython-312
      creating build/lib.linux-aarch64-cpython-312/webrtc_noise_gain
      copying webrtc_noise_gain/__init__.py -> build/lib.linux-aarch64-cpython-312/webrtc_noise_gain
      running build_ext
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/homeassistant/.pyenv/versions/3.12.1/include/python3.12 -c -std=c++17 flagcheck.cpp -o flagcheck.o -std=c++17
      gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
      compilation terminated.
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/homeassistant/.pyenv/versions/3.12.1/include/python3.12 -c -std=c++17 flagcheck.cpp -o flagcheck.o -std=c++14
      gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
      compilation terminated.
      gcc -fno-strict-overflow -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -I/home/homeassistant/.pyenv/versions/3.12.1/include/python3.12 -c -std=c++17 flagcheck.cpp -o flagcheck.o -std=c++11
      gcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory
      compilation terminated.
      Traceback (most recent call last):
        File "/home/homeassistant/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/homeassistant/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/homeassistant/.pyenv/versions/3.12.1/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 563, in <module>
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 103, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ecc9vjza/normal/lib/python3.12/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build.py", line 131, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/dist.py", line 963, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/command/build_ext.py", line 89, in run
          _build_ext.run(self)
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/setuptools/_distutils/command/build_ext.py", line 345, in run
          self.build_extensions()
        File "<string>", line 511, in build_extensions
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/pybind11/setup_helpers.py", line 283, in build_extensions
          ext.cxx_std = auto_cpp_level(self.compiler)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-ecc9vjza/overlay/lib/python3.12/site-packages/pybind11/setup_helpers.py", line 266, in auto_cpp_level
          raise RuntimeError(msg)
      RuntimeError: Unsupported compiler -- at least C++11 support is needed!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for webrtc-noise-gain
Failed to build webrtc-noise-gain

Here are the errors shown in HA:
image
image

I'm facing that issue on fresh install as well, meaning that the current startup scripts are not in position to find all the necessary deps.
I'm currently installing g++ again to add webrtc-noise-gain back to my py env and check if the issue vanishes or not

@Joulinar
Copy link
Collaborator

See #6931

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Enhancement 💨 Solution available 🥂 Definite solution has been done
Projects
None yet
Milestone
v8.15
Development

No branches or pull requests
4 participants
@cazzoo @MichaIng @maury77 @Joulinar