Support Python 3.12 #6539

JamesClarke7283 · 2024-06-24T13:20:10Z

Summary

This PR updates the pinned dependencies in the pyproject.toml to support python3.12. this is a first draft.

If we are making use of asyncio in some capacity, it might improve performance a fair amount in some use cases, as a added bonus.

Related Issues / Discussions

Will likely fix #6214
Will also likely fix #5674 although, we may want to add scripts which update the pinned versions to the lowest version that x python version supports, as long as the version is equal to or greater than whats already in the dependencies.

QA Instructions

# Setting up the environment
export INVOKEAI_ROOT=~/invokeai
mkdir $INVOKEAI_ROOT
cd $INVOKEAI_ROOT
python3.12 -m venv .venv --prompt InvokeAI
source .venv/bin/activate

# Cloning Invoke
cd ~/Downloads
git clone https://github.com/JamesClarke7283/InvokeAI --depth 1

# Installing dependencies
cd InvokeAI
pip install -e . --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121 # if using cuda
cd $INVOKEAI_ROOT
deactivate && source .venv/bin/activate

# build the frontend
cd ~/Downloads/InvokeAI/invokeai/frontend/web/
pnpm i
pnpm build

# run the program
cd $INVOKEAI_ROOT
invokeai-web

Merge Plan

Should be fairly simple to merge, as long as the tests pass, its mostly only the pyproject.toml that needs editing at the moment.

Checklist

Update pyproject.toml with dependencies that support 3.12.
Check if it installs correctly.
Make changes to automatic install script to support 3.12.
Add Python3.12 build github workflow.

lstein

Unfortunately I do not have a system with both 3.12 and a CUDA GPU, so I can't test this properly. It would be good to get testing on both CUDA and MPS systems.

I did however update the installer and its readme so that it will allow python 3.12.

leder11011 · 2024-06-25T08:08:14Z

Hello @lstein, I have both - rtx 4090 - Python 3.12 What should I be testing? Lincoln Stein ***@***.***> schrieb am Di., 25. Juni 2024, 02:56:

…

***@***.**** commented on this pull request. Unfortunately I do not have a system with both 3.12 and a CUDA GPU, so I can't test this properly. It would be good to get testing on both CUDA and MPS systems. I did however update the installer and its readme so that it will allow python 3.12. — Reply to this email directly, view it on GitHub <#6539 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNARBAPKKKB6LFPLC4ROLZJC52VAVCNFSM6AAAAABJZ2IZGCVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMZXGA3DIOBXGE> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

leder11011 · 2024-06-25T13:13:44Z

@JamesClarke7283 @lstein
Installing your repo and main branch I get the following error running ´./install.sh.in´:

ERROR: Ignored the following versions that require a different python version: 2.2.4.5 Requires-Python <3.11,>=3.9; 2.2.4.6 Requires-Python <3.11,>=3.9; 2.2.4.7 Requires-Python <3.11,>=3.9; 2.2.5 Requires-Python <3.11,>=3.9; 2.3.0 Requires-Python <3.11,>=3.9; 2.3.0a0 Requires-Python <3.11,>=3.9; 2.3.0a1 Requires-Python <3.11,>=3.9; 2.3.0a2 Requires-Python <3.11,>=3.9; 2.3.0a3 Requires-Python <3.11,>=3.9; 2.3.0rc3 Requires-Python <3.11,>=3.9; 2.3.0rc4 Requires-Python <3.11,>=3.9; 2.3.0rc5 Requires-Python <3.11,>=3.9; 2.3.0rc6 Requires-Python <3.11,>=3.9; 2.3.0rc7 Requires-Python <3.11,>=3.9; 2.3.1 Requires-Python <3.11,>=3.9; 2.3.1.post1 Requires-Python <3.11,>=3.9; 2.3.1.post2 Requires-Python <3.11,>=3.9; 2.3.1rc4 Requires-Python <3.11,>=3.9; 2.3.2 Requires-Python <3.11,>=3.9; 2.3.2.post1 Requires-Python <3.11,>=3.9; 2.3.3 Requires-Python <3.11,>=3.9; 2.3.3rc1 Requires-Python <3.11,>=3.9; 2.3.4 Requires-Python <3.11,>=3.9; 2.3.4.post1 Requires-Python <3.11,>=3.9; 2.3.4a0 Requires-Python <3.11,>=3.9; 2.3.4rc1 Requires-Python <3.11,>=3.9; 2.3.5 Requires-Python <3.11,>=3.9; 2.3.5.post1 Requires-Python <3.11,>=3.9; 2.3.5.post2 Requires-Python <3.11,>=3.9; 2.3.5rc1 Requires-Python <3.11,>=3.9; 3.0.0 Requires-Python <3.11,>=3.9; 3.0.1 Requires-Python <3.12,>=3.9; 3.0.1.post1 Requires-Python <3.12,>=3.9; 3.0.1.post2 Requires-Python <3.12,>=3.9; 3.0.1.post3 Requires-Python <3.12,>=3.9; 3.0.1rc1 Requires-Python <3.11,>=3.9; 3.0.1rc2 Requires-Python <3.12,>=3.9; 3.0.2 Requires-Python <3.12,>=3.9; 3.0.2.post1 Requires-Python <3.12,>=3.9; 3.0.2a1 Requires-Python <3.12,>=3.9; 3.0.2rc1 Requires-Python <3.12,>=3.9; 3.1.0 Requires-Python <3.12,>=3.9; 3.1.1 Requires-Python <3.12,>=3.9; 3.1.1rc1 Requires-Python <3.12,>=3.9; 3.2.0 Requires-Python <3.12,>=3.9; 3.3.0 Requires-Python <3.12,>=3.9; 3.3.0.post1 Requires-Python <3.12,>=3.9; 3.3.0.post2 Requires-Python <3.12,>=3.9; 3.3.0.post3 Requires-Python <3.12,>=3.9; 3.4.0 Requires-Python <3.12,>=3.10; 3.4.0.post1 Requires-Python <3.12,>=3.10; 3.4.0.post2 Requires-Python <3.12,>=3.10; 3.4.0rc2 Requires-Python <3.12,>=3.10; 3.4.0rc3 Requires-Python <3.12,>=3.10; 3.4.0rc4 Requires-Python <3.12,>=3.10; 3.5.0 Requires-Python <3.12,>=3.10; 3.5.0rc1 Requires-Python <3.12,>=3.10; 3.5.0rc2 Requires-Python <3.12,>=3.10; 3.5.0rc3 Requires-Python <3.12,>=3.10; 3.5.1 Requires-Python <3.12,>=3.10; 3.6.0 Requires-Python <3.12,>=3.10; 3.6.0rc1 Requires-Python <3.12,>=3.10; 3.6.0rc2 Requires-Python <3.12,>=3.10; 3.6.0rc3 Requires-Python <3.12,>=3.10; 3.6.0rc4 Requires-Python <3.12,>=3.10; 3.6.0rc5 Requires-Python <3.12,>=3.10; 3.6.0rc6 Requires-Python <3.12,>=3.10; 3.6.1 Requires-Python <3.12,>=3.10; 3.6.2 Requires-Python <3.12,>=3.10; 3.6.3 Requires-Python <3.12,>=3.10; 3.6.3rc1 Requires-Python <3.12,>=3.10; 3.7.0 Requires-Python <3.12,>=3.10; 4.0.0 Requires-Python <3.12,>=3.10; 4.0.0rc1 Requires-Python <3.12,>=3.10; 4.0.0rc2 Requires-Python <3.12,>=3.10; 4.0.0rc4 Requires-Python <3.12,>=3.10; 4.0.0rc5 Requires-Python <3.12,>=3.10; 4.0.0rc6 Requires-Python <3.12,>=3.10; 4.0.1 Requires-Python <3.12,>=3.10; 4.0.2 Requires-Python <3.12,>=3.10; 4.0.3 Requires-Python <3.12,>=3.10; 4.0.4 Requires-Python <3.12,>=3.10; 4.1.0 Requires-Python <3.12,>=3.10; 4.2.0 Requires-Python <3.12,>=3.10; 4.2.0a1 Requires-Python <3.12,>=3.10; 4.2.0a2 Requires-Python <3.12,>=3.10; 4.2.0a3 Requires-Python <3.12,>=3.10; 4.2.0a4 Requires-Python <3.12,>=3.10; 4.2.0b1 Requires-Python <3.12,>=3.10; 4.2.0b2 Requires-Python <3.12,>=3.10; 4.2.1 Requires-Python <3.12,>=3.10; 4.2.2 Requires-Python <3.12,>=3.10; 4.2.2.post1 Requires-Python <3.12,>=3.10; 4.2.3 Requires-Python <3.12,>=3.10; 4.2.4 Requires-Python <3.12,>=3.10
ERROR: Could not find a version that satisfies the requirement invokeai[onnx-cuda,xformers] (from versions: none)
ERROR: No matching distribution found for invokeai[onnx-cuda,xformers]

leder11011 · 2024-06-27T18:58:19Z

I followd "QA instructions" as above until the step:
pip install -e . --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121 # if using cuda
which gave the following error:

  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      + meson setup /tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc /tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc/.mesonpy-zw_g60wh -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc/.mesonpy-zw_g60wh/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.1
      Source dir: /tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc
      Build dir: /tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc/.mesonpy-zw_g60wh
      Build type: native build
      
      ../meson.build:7:11: ERROR: Command `/tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc/skimage/_build_utils/version.py` failed with status 127.
      
      A full log can be found at /tmp/pip-install-7g5mbpyd/scikit-image_f88d1b4fcbd94cc79792b8c4f97980fc/.mesonpy-zw_g60wh/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

JamesClarke7283 · 2024-07-05T08:54:37Z

Last i checked, some dependencies did not have versions supporting 3.12 as wheels are not build for the older versions. I will give it another try now...

JamesClarke7283 · 2024-07-05T09:20:39Z

I was able to get the WebUI up by using these commands:

export INVOKEAI_ROOT=~/invokeai
mkdir $INVOKEAI_ROOT
cd $INVOKEAI_ROOT
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e git+https://github.com/JamesClarke7283/InvokeAI#egg=InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121

deactivate && source .venv/bin/activate
cd $INVOKEAI_ROOT/.venv/src/invokeai/invokeai/frontend/web
pnpm i
pnpm build
cd $INVOKEAI_ROOT

invokeai-web

Manual dev build works!

hipsterusername · 2024-07-05T11:52:04Z

Think we need testing on MPS still - Discord might be a source of good recruits there

leder11011 · 2024-07-05T14:28:02Z

I was able to get the WebUI up by using these commands:

export INVOKEAI_ROOT=~/invokeai
mkdir $INVOKEAI_ROOT
cd $INVOKEAI_ROOT
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e git+https://github.com/JamesClarke7283/InvokeAI#egg=InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121

deactivate && source .venv/bin/activate
cd $INVOKEAI_ROOT/.venv/src/invokeai/invokeai/frontend/web
pnpm i
pnpm build
cd $INVOKEAI_ROOT

invokeai-web

Manual dev build works!

I still have an issue:

  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      + meson setup /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497 /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.1
      Source dir: /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497
      Build dir: /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w
      Build type: native build
      
      ../meson.build:7:11: ERROR: Command `/tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/skimage/_build_utils/version.py` failed with status 127.
      
      A full log can be found at /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

JamesClarke7283 · 2024-07-08T06:58:08Z

I was able to get the WebUI up by using these commands:

export INVOKEAI_ROOT=~/invokeai
mkdir $INVOKEAI_ROOT
cd $INVOKEAI_ROOT
python3.12 -m venv .venv
source .venv/bin/activate
pip install -e git+https://github.com/JamesClarke7283/InvokeAI#egg=InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121

deactivate && source .venv/bin/activate
cd $INVOKEAI_ROOT/.venv/src/invokeai/invokeai/frontend/web
pnpm i
pnpm build
cd $INVOKEAI_ROOT

invokeai-web

Manual dev build works!

I still have an issue:

  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [10 lines of output]
      + meson setup /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497 /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md --native-file=/tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w/meson-python-native-file.ini
      The Meson build system
      Version: 1.4.1
      Source dir: /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497
      Build dir: /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w
      Build type: native build
      
      ../meson.build:7:11: ERROR: Command `/tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/skimage/_build_utils/version.py` failed with status 127.
      
      A full log can be found at /tmp/pip-install-jmrxj3cn/scikit-image_d9e867d53aee461c910340ed46908497/.mesonpy-2stzgn2w/meson-logs/meson-log.txt
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

I am going to need the full log file mentioned in that error output. Make sure you have all system dependencies installed as well.

JamesClarke7283 · 2024-07-08T07:04:32Z

@ebr @hipsterusername @blessedcoolant Can you approve the workflows to run please?

I don't see any issues with the manual build, installer build i have not tested yet with it but at least if these pass i know its mostly sound and can proceed with further testing.

leder11011 · 2024-07-08T07:33:04Z

@james Clark My manual build fails with above error. James Clarke ***@***.***> schrieb am Mo., 8. Juli 2024, 09:04:

…

@ebr <https://github.com/ebr> @hipsterusername <https://github.com/hipsterusername> @blessedcoolant <https://github.com/blessedcoolant> Can you approve the workflows to run please? I don't see any issues with the manual build, installer build i have not tested yet with it but at least if these pass i know its mostly sound. — Reply to this email directly, view it on GitHub <#6539 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNARBQ6VELTGLZIY7QPCLZLI2ZNAVCNFSM6AAAAABJZ2IZGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMJTGE4TOMJTGE> . You are receiving this because you commented.Message ID: ***@***.***>

ebr · 2024-07-08T11:55:41Z

We are pinning scikit-image=~0.21.1, which does not support Python 3.12. @JamesClarke7283 - you might have a later version cached, which makes the build work for you, but not for @leder11011 . You might be able to test this with pip --no-cache --force-reinstall flags. In any case, we need to find out why pinning scikit-image to this version was necessary, and perform thorough regression testing on all platforms with a newer version.

leder11011 · 2024-07-08T17:25:02Z

We are pinning scikit-image=~0.21.1, which does not support Python 3.12. @JamesClarke7283 - you might have a later version cached, which makes the build work for you, but not for @leder11011 . You might be able to test this with pip --no-cache --force-reinstall flags. In any case, we need to find out why pinning scikit-image to this version was necessary, and perform thorough regression testing on all platforms with a newer version.

I tried pip --no-cache --force-reinstall ... on @JamesClarke7283 repo with current version of main branch: scikit_image~=0.21.0 is used and error persists.

leder11011 · 2024-07-08T17:35:37Z

another try with scikit-image~=0.24.0 and

pip install -e git+https://github.com/JamesClarke7283/InvokeAI\#egg=InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121 --no-cache --force-reinstall

complains about scikit-image, too.

The full log mentioned in directory tmp is deleted: How do I access the logs?

psychedelicious · 2024-07-09T00:46:16Z

Two issues with the GH actions tests:

The python tests are failing because the pip package is currently flagged as compatible with 3.10 and 3.11 only.
The test matrix doesn't include 3.12.

We'll need to point the actions at this branch and add 3.12 to the test matrix workflow. Not sure how to sequence things to make this work... Suppose we could change the python version reqs in pyproject.toml and release to pypi.

infinity0 · 2024-07-16T18:09:45Z

Please merge this PR soon, Debian Testing's default Python is now 3.12 and 3.11 will get automatically uninstalled from users' machines when they next upgrade.

fat-tire · 2024-07-16T22:13:34Z

Four days ago, the nvidia/cuda latest docker image was updated to default to python 3.12 as well, which immediately broke existing 3.10-based venv installations.

pip install -e git+https://github.com/JamesClarke7283/InvokeAI#egg=InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121 successfully reinstalled all the pip packages needed to get 4.2.5 working (actually used cu124) again, but won't auto-upgrade to 4.2.6p1 via ./invokeai.sh option 9 until an official package is made that supports 3.12.

I could update manually on top of the current 4.25 but figure I can wait now that everything is running again. But yeah, I have a feeling this will become more of an issue pretty soon.

Update-- I did have to rebuild patchmatch per a notice when running. Just had to import the -dev depencines and then import into python and it built and the notice went away.

hipsterusername · 2024-07-16T23:26:45Z

Thanks for the nudges all - we’ll get this in soon.

leder11011 · 2024-07-17T05:02:34Z

@kent I object Kent Keirsey ***@***.***> schrieb am Mi., 17. Juli 2024, 01:27:

…

Thanks for the nudges all - we’ll get this in soon. — Reply to this email directly, view it on GitHub <#6539 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNARDXXNHPUC7CNWTY7XDZMWT4ZAVCNFSM6AAAAABJZ2IZGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRHE3TSNBTGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

hipsterusername · 2024-07-17T11:34:59Z

@kent I object Kent Keirsey @.***> schrieb

I’m not proposing that we will merge it in without changes/fixing. Just that we recognize there are compelling reasons to evaluate how to solve any issues people will face and get moving to 3.12

This may not be a simple process.

hipsterusername · 2024-07-17T12:29:11Z

On review - Understanding that there are some emerging inconveniences that would make getting 3.12 migrated to, I think the rub here is going to be ensuring that all deep-down dependencies play well with it, and it will absorb a significant amount of time for regression testing and updating any language changes.

This will happen - Eventually. But for the time being, we may rely on users to use some simple workarounds for the default python versions getting installed. E.g., installing 11.

psychedelicious · 2024-07-17T13:00:28Z

There's no pressing need to support 3.12 for the Invoke application itself. We don't gain anything meaningful. The main reasons to support 3.12 are the user-facing installation road bumps described in this issue.

Workarounds

Until we get python 3.12 support in, there's a fairly straightforward and conventional workaround to get the installer to use python 3.11 on a system where python 3.12 is installed.

Ensure python 3.11 is before python 3.12 in your `PATH`

You need the python 3.11 location to be first in the PATH environment variable, ahead of python 3.12's location:

On macOS and Linux, this is very straightforward via terminal. Plenty of guides if you search.
On Windows, there's a GUI somewhere in Control Panel. Again, plenty of guides showing how to edit the env var.

If you don't want to edit the env var yourself, you could use a python version manager:

On macOS and Linux, pyenv is the de-facto util.
On Windows, use this fork pyenv-win instead.
Follow the instructions to set up pyenv and install python 3.11 with it, setting it as your system python version.

Now that python 3.11 is higher in precedence in your PATH, the installer will automatically use it. Once installed, you can revert your PATH changes (manually or via pyenv). Invoke will continue to use the python with which it was installed.

Docker image

I believe we'll need to update the docker config to fix the issue with the latest image defaulting to 3.12.

Installer CLI arg

We could add a --python-path CLI arg to the install scripts, so you can directly specify the python you want to use. Probably a good idea.

leder11011 · 2024-07-17T13:10:03Z

Thank you @hipsterusername and @psychedelicious

As I commented here I do not have the option on manjaro to do an additional pyenv install of python 3.11:

#5674 (comment)

I am relying on fixing this pull request - additionally implementing a workaround for an additional 3.11 installation is fine with me, though.

psychedelicious · 2024-07-17T14:09:58Z

@leder11011 Hmm, I use pyenv to install on Ubuntu, have multiple versions installed. May be some other issue you are running into - pyenv definitely works in general.

fat-tire · 2024-07-18T01:17:47Z

Docker image

I believe we'll need to update the docker config to fix the issue with the latest image defaulting to 3.12.

Does it now do this? I'm running 4.2.5 successfully in the new "nvidia/cuda" container (24.04) (re-installed packages for updated 3.12 venv w/pip install per above), but it won't self-update to 4.2.6.post1 due to the <3.12 requirement on the update packages.

psychedelicious · 2024-07-18T01:55:35Z

@fat-tire I mean we probably need to update our docker do not use the latest image and instead use an image that provides 3.11.

leder11011 · 2024-07-18T02:13:56Z

@leder11011 Hmm, I use pyenv to install on Ubuntu, have multiple versions installed. May be some other issue you are running into - pyenv definitely works in general.

It is easily possible that the error lies between the keyboard and the seat as I am no proficient python developer - but shouldn't we investigate in the first place?

ebr · 2024-07-19T03:56:49Z

Our image is based on ubuntu23.04 which ships with python3.11. python3.12 as default was introduced in 24.04, and we are not in a hurry to upgrade until we can support it, but it shouldn't be a complex change. doesn't need to be coupled with this PR.

psychedelicious · 2024-07-19T05:27:00Z

It is easily possible that the error lies between the keyboard and the seat as I am no proficient python developer - but shouldn't we investigate in the first place?

The error you got was:
/home/leder/invokeai/.venv/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

Maybe you need to install a devel package for python to get the shared libraries? Sorry, I'm not sure how to address this.

leder11011 · 2024-07-19T20:08:08Z

Thank you! I will have a look. psychedelicious ***@***.***> schrieb am Fr., 19. Juli 2024, 07:27:

…

It is easily possible that the error lies between the keyboard and the seat as I am no proficient python developer - but shouldn't we investigate in the first place? The error you got was: /home/leder/invokeai/.venv/bin/python: error while loading shared libraries: libpython3.11.so.1.0: cannot open shared object file: No such file or directory Maybe you need to install a devel package for python to get the shared libraries? Sorry, I'm not sure how to address this. — Reply to this email directly, view it on GitHub <#6539 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZNARAWFHPVI7LHZNZGS7TZNCPTVAVCNFSM6AAAAABJZ2IZGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZYGIZDEMBWHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

leder11011 · 2024-07-20T03:17:25Z

Now I have an unhappy version mix of 3.11 and 3.12:

   ~/InvokeAI-Installer  /home/leder/invokeai/.venv/bin/pip install --require-virtualenv --force-reinstall --use-pep517 'invokeai[xformers,onnx-cuda]==4.2.6post1'                     INT ✘    3.11.6  

ERROR: Ignored the following versions that require a different python version: 2.2.4.5 Requires-Python <3.11,>=3.9; 2.2.4.6 Requires-Python <3.11,>=3.9; 2.2.4.7 Requires-Python <3.11,>=3.9; 2.2.5 Requires-Python <3.11,>=3.9; 2.3.0 Requires-Python <3.11,>=3.9; 2.3.0a0 Requires-Python <3.11,>=3.9; 2.3.0a1 Requires-Python <3.11,>=3.9; 2.3.0a2 Requires-Python <3.11,>=3.9; 2.3.0a3 Requires-Python <3.11,>=3.9; 2.3.0rc3 Requires-Python <3.11,>=3.9; 2.3.0rc4 Requires-Python <3.11,>=3.9; 2.3.0rc5 Requires-Python <3.11,>=3.9; 2.3.0rc6 Requires-Python <3.11,>=3.9; 2.3.0rc7 Requires-Python <3.11,>=3.9; 2.3.1 Requires-Python <3.11,>=3.9; 2.3.1.post1 Requires-Python <3.11,>=3.9; 2.3.1.post2 Requires-Python <3.11,>=3.9; 2.3.1rc4 Requires-Python <3.11,>=3.9; 2.3.2 Requires-Python <3.11,>=3.9; 2.3.2.post1 Requires-Python <3.11,>=3.9; 2.3.3 Requires-Python <3.11,>=3.9; 2.3.3rc1 Requires-Python <3.11,>=3.9; 2.3.4 Requires-Python <3.11,>=3.9; 2.3.4.post1 Requires-Python <3.11,>=3.9; 2.3.4a0 Requires-Python <3.11,>=3.9; 2.3.4rc1 Requires-Python <3.11,>=3.9; 2.3.5 Requires-Python <3.11,>=3.9; 2.3.5.post1 Requires-Python <3.11,>=3.9; 2.3.5.post2 Requires-Python <3.11,>=3.9; 2.3.5rc1 Requires-Python <3.11,>=3.9; 3.0.0 Requires-Python <3.11,>=3.9; 3.0.1 Requires-Python <3.12,>=3.9; 3.0.1.post1 Requires-Python <3.12,>=3.9; 3.0.1.post2 Requires-Python <3.12,>=3.9; 3.0.1.post3 Requires-Python <3.12,>=3.9; 3.0.1rc1 Requires-Python <3.11,>=3.9; 3.0.1rc2 Requires-Python <3.12,>=3.9; 3.0.2 Requires-Python <3.12,>=3.9; 3.0.2.post1 Requires-Python <3.12,>=3.9; 3.0.2a1 Requires-Python <3.12,>=3.9; 3.0.2rc1 Requires-Python <3.12,>=3.9; 3.1.0 Requires-Python <3.12,>=3.9; 3.1.1 Requires-Python <3.12,>=3.9; 3.1.1rc1 Requires-Python <3.12,>=3.9; 3.2.0 Requires-Python <3.12,>=3.9; 3.3.0 Requires-Python <3.12,>=3.9; 3.3.0.post1 Requires-Python <3.12,>=3.9; 3.3.0.post2 Requires-Python <3.12,>=3.9; 3.3.0.post3 Requires-Python <3.12,>=3.9; 3.4.0 Requires-Python <3.12,>=3.10; 3.4.0.post1 Requires-Python <3.12,>=3.10; 3.4.0.post2 Requires-Python <3.12,>=3.10; 3.4.0rc2 Requires-Python <3.12,>=3.10; 3.4.0rc3 Requires-Python <3.12,>=3.10; 3.4.0rc4 Requires-Python <3.12,>=3.10; 3.5.0 Requires-Python <3.12,>=3.10; 3.5.0rc1 Requires-Python <3.12,>=3.10; 3.5.0rc2 Requires-Python <3.12,>=3.10; 3.5.0rc3 Requires-Python <3.12,>=3.10; 3.5.1 Requires-Python <3.12,>=3.10; 3.6.0 Requires-Python <3.12,>=3.10; 3.6.0rc1 Requires-Python <3.12,>=3.10; 3.6.0rc2 Requires-Python <3.12,>=3.10; 3.6.0rc3 Requires-Python <3.12,>=3.10; 3.6.0rc4 Requires-Python <3.12,>=3.10; 3.6.0rc5 Requires-Python <3.12,>=3.10; 3.6.0rc6 Requires-Python <3.12,>=3.10; 3.6.1 Requires-Python <3.12,>=3.10; 3.6.2 Requires-Python <3.12,>=3.10; 3.6.3 Requires-Python <3.12,>=3.10; 3.6.3rc1 Requires-Python <3.12,>=3.10; 3.7.0 Requires-Python <3.12,>=3.10; 4.0.0 Requires-Python <3.12,>=3.10; 4.0.0rc1 Requires-Python <3.12,>=3.10; 4.0.0rc2 Requires-Python <3.12,>=3.10; 4.0.0rc4 Requires-Python <3.12,>=3.10; 4.0.0rc5 Requires-Python <3.12,>=3.10; 4.0.0rc6 Requires-Python <3.12,>=3.10; 4.0.1 Requires-Python <3.12,>=3.10; 4.0.2 Requires-Python <3.12,>=3.10; 4.0.3 Requires-Python <3.12,>=3.10; 4.0.4 Requires-Python <3.12,>=3.10; 4.1.0 Requires-Python <3.12,>=3.10; 4.2.0 Requires-Python <3.12,>=3.10; 4.2.0a1 Requires-Python <3.12,>=3.10; 4.2.0a2 Requires-Python <3.12,>=3.10; 4.2.0a3 Requires-Python <3.12,>=3.10; 4.2.0a4 Requires-Python <3.12,>=3.10; 4.2.0b1 Requires-Python <3.12,>=3.10; 4.2.0b2 Requires-Python <3.12,>=3.10; 4.2.1 Requires-Python <3.12,>=3.10; 4.2.2 Requires-Python <3.12,>=3.10; 4.2.2.post1 Requires-Python <3.12,>=3.10; 4.2.3 Requires-Python <3.12,>=3.10; 4.2.4 Requires-Python <3.12,>=3.10; 4.2.5 Requires-Python <3.12,>=3.10; 4.2.6 Requires-Python <3.12,>=3.10; 4.2.6.post1 Requires-Python <3.12,>=3.10; 4.2.6a1 Requires-Python <3.12,>=3.10; 4.2.6rc1 Requires-Python <3.12,>=3.10
ERROR: Could not find a version that satisfies the requirement invokeai==4.2.6post1 (from versions: none)
ERROR: No matching distribution found for invokeai==4.2.6post1

ebr · 2024-07-25T17:17:27Z

@JamesClarke7283 I'm unable to build this due to dependency issues - pip can't find torch for some reason. Could you please sync your fork with main and resolve the conflict in pyproject.toml, and I will give it another try?
Thank you

JamesClarke7283 · 2024-08-15T07:51:43Z

@JamesClarke7283 I'm unable to build this due to dependency issues - pip can't find torch for some reason. Could you please sync your fork with main and resolve the conflict in pyproject.toml, and I will give it another try? Thank you

Ive resolved the conflicts.

leder11011 · 2024-08-27T01:19:09Z

seems as if dependency pytorch will directly jump to python 3.13 support:
pytorch/pytorch#130249

keturn · 2024-08-30T01:16:24Z

Ubuntu 24.04.1 LTS is now out. This triggers existing Ubuntu 22.04 LTS systems to prompt an upgrade, and the default choice for a new LTS installation will be 24.04 (if it wasn't already).

[as ebr pointed out earlier, this Ubuntu release uses Python 3.12. The 23.04 used by the docker image is seven months past its EOL now.]

fat-tire · 2024-08-30T04:30:34Z

Got this to update from 4.2.7post1 to 4.2.9rc1 w/python3.12...

First I edited:
/invokeai/.venv/lib/python3.12/site-packages/invokeai/frontend/install/invokeai_update.py so that in the pip install line near the bottom add --ignore-requires-python to ignore the dependency on 3.10 or whatever. Then if you update via the menu, you'll probably have a mediapipe compatability issue, which you can fix by adding --no-deps as well.

Then python3 /invokeai/.venv/lib/python3.12/site-packages/invokeai/frontend/install/invokeai_update.py to do the update.

Downloaded the Flux.1 schnell quantized model, installed bitsandbytes & sentencepiece, and it seems to work fine. [updated: had a memory error because I forgot to kill an llm I had running. It generated via flux no problemo!]

JamesClarke7283 · 2024-09-03T08:42:14Z

Feel free to work on this PR, i won't be maintaining it, just a FYI.
It works well for me, we just need the auto installer to work i guess.

leder11011 · 2024-10-25T02:36:09Z

Are you serious: works on my machine?

sonarcloud · 2024-10-25T21:09:01Z

Quality Gate failed

Failed conditions
C Maintainability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

Updated pinned dependencues and enabled python 3.12

a4cc349

JamesClarke7283 requested review from lstein, blessedcoolant and hipsterusername as code owners June 24, 2024 13:20

JamesClarke7283 marked this pull request as draft June 24, 2024 13:20

github-actions bot added Root python-deps PRs that change python dependencies labels Jun 24, 2024

Revert diffusers dep back to 0.27.2 due to API change

3b3a0d1

JamesClarke7283 marked this pull request as ready for review June 24, 2024 14:40

updated installer to allow 3.12

3defd5a

lstein requested a review from ebr as a code owner June 25, 2024 00:54

github-actions bot added the installer PRs that change the installer label Jun 25, 2024

lstein reviewed Jun 25, 2024

View reviewed changes

ebr self-assigned this Jun 25, 2024

Merge branch 'main' into main

30cc1c9

Merge branch 'main' into main

669432f

infinity0 mentioned this pull request Jul 25, 2024

[bug]: WebUI client-side (JS) does not garbage collect memory until user refreshes the page #6677

Closed

1 task

Merge branch 'main' into main

780915a

JamesClarke7283 and others added 2 commits August 15, 2024 09:00

Fix by adding back spandrel dependency

fcd00c7

Merge branch 'main' into main

c100814

leder11011 mentioned this pull request Oct 25, 2024

[bug]: GIT installer.sh.in and python 3.12 #7194

Closed

1 task

Support Python 3.12 #6539

Are you sure you want to change the base?

Support Python 3.12 #6539

Conversation

JamesClarke7283 commented Jun 24, 2024 • edited Loading

Summary

Related Issues / Discussions

QA Instructions

Merge Plan

Checklist

lstein left a comment

Choose a reason for hiding this comment

leder11011 commented Jun 25, 2024 via email

leder11011 commented Jun 25, 2024

leder11011 commented Jun 27, 2024

JamesClarke7283 commented Jul 5, 2024

JamesClarke7283 commented Jul 5, 2024 • edited Loading

hipsterusername commented Jul 5, 2024

leder11011 commented Jul 5, 2024

JamesClarke7283 commented Jul 8, 2024 • edited Loading

JamesClarke7283 commented Jul 8, 2024 • edited Loading

leder11011 commented Jul 8, 2024 via email

ebr commented Jul 8, 2024

leder11011 commented Jul 8, 2024

leder11011 commented Jul 8, 2024

psychedelicious commented Jul 9, 2024

infinity0 commented Jul 16, 2024

fat-tire commented Jul 16, 2024 • edited Loading

hipsterusername commented Jul 16, 2024

leder11011 commented Jul 17, 2024 via email

hipsterusername commented Jul 17, 2024

hipsterusername commented Jul 17, 2024

psychedelicious commented Jul 17, 2024

Workarounds

Ensure python 3.11 is before python 3.12 in your PATH

Docker image

Installer CLI arg

leder11011 commented Jul 17, 2024

psychedelicious commented Jul 17, 2024

fat-tire commented Jul 18, 2024 • edited Loading

psychedelicious commented Jul 18, 2024

leder11011 commented Jul 18, 2024

ebr commented Jul 19, 2024 • edited Loading

psychedelicious commented Jul 19, 2024

leder11011 commented Jul 19, 2024 via email

leder11011 commented Jul 20, 2024

ebr commented Jul 25, 2024

JamesClarke7283 commented Aug 15, 2024

leder11011 commented Aug 27, 2024

keturn commented Aug 30, 2024 • edited Loading

fat-tire commented Aug 30, 2024 • edited Loading

JamesClarke7283 commented Sep 3, 2024

leder11011 commented Oct 25, 2024

sonarcloud bot commented Oct 25, 2024

Quality Gate failed

JamesClarke7283 commented Jun 24, 2024 •

edited

Loading

JamesClarke7283 commented Jul 5, 2024 •

edited

Loading

JamesClarke7283 commented Jul 8, 2024 •

edited

Loading

JamesClarke7283 commented Jul 8, 2024 •

edited

Loading

fat-tire commented Jul 16, 2024 •

edited

Loading

Ensure python 3.11 is before python 3.12 in your `PATH`

fat-tire commented Jul 18, 2024 •

edited

Loading

ebr commented Jul 19, 2024 •

edited

Loading

keturn commented Aug 30, 2024 •

edited

Loading

fat-tire commented Aug 30, 2024 •

edited

Loading