poetry init Unable to Find Package During Interactive Dependency Addition

[hakuna-max]

Description

When using the poetry init command to create a new project, the interactive dependency addition step fails to find packages (e.g., numpy). However, the poetry add command works as expected for the same package. The issue appears to be related to how poetry init handles package search and HTTP requests.

Workarounds

After completing the poetry init process, dependencies can be added using the poetry add command. For example:poetry add numpy. This bypasses the issue with the interactive dependency search step.

Poetry Installation Method

install.python-poetry.org

Operating System

Windows 11 24H2

Poetry Version

1.8.4

Poetry Configuration

cache-dir = "C:\\Users\\rolfz\\AppData\\Local\\pypoetry\\Cache"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs"  # C:\Users\rolfz\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "nt"

Paths:
        data = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        include = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Include"
        platinclude = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Include"
        platlib = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib\site-packages"
        platstdlib = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib"
        purelib = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib\site-packages"
        scripts = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Scripts"
        stdlib = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib"

Variables:
        BINDIR = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        BINLIBDEST = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib"
        EXE = ".exe"
        EXT_SUFFIX = ".cp312-win_amd64.pyd"
        INCLUDEPY = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Include"
        LIBDEST = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0\Lib"
        TZPATH = ""
        VERSION = "312"
        VPATH = "..\.."
        abiflags = ""
        base = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        exec_prefix = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        installed_base = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        installed_platbase = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        platbase = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        platlibdir = "DLLs"
        prefix = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        projectbase = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        py_version = "3.12.0"
        py_version_nodot = "312"
        py_version_nodot_plat = "312"
        py_version_short = "3.12"
        srcdir = "C:\Users\rolfz\.pyenv\pyenv-win\versions\3.12.0"
        userbase = "C:\Users\rolfz\AppData\Roaming\Python"

Example pyproject.toml

The issue occurs during the poetry init process, so no pyproject.toml file is generated yet.


# Example pyproject.toml that would have been generated if the issue did not occur
[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["peachy <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
numpy = "^1.26.0"  # This dependency cannot be added during the `poetry init` process


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

PS D:\hub\test> poetry init -vvv

This command will guide you through creating your pyproject.toml config.

Package name [test]:
Version [0.1.0]:
Description []:
Author [peachy <[email protected]>, n to skip]:
License []:
Loading configuration file C:\Users\rolfz\AppData\Roaming\pypoetry\config.toml
Compatible Python versions [^3.13]:  ^3.12

Would you like to define your main dependencies interactively? (yes/no) [yes]
You can specify a package in the following forms:
  - A single name (requests): this will search for matches on PyPI
  - A name and a constraint (requests@^2.23.0)
  - A git url (git+https://github.com/python-poetry/poetry.git)
  - A git url with a revision (git+https://github.com/python-poetry/poetry.git#develop)
  - A file path (../my-package/my-package.whl)
  - A directory (../my-package/)
  - A url (https://example.com/packages/my-package-0.1.0.tar.gz)

Package to add or search for (leave blank to skip): numpy
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search?q=numpy HTTP/11" 301 213
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search/?q=numpy HTTP/11" 200 23878
Unable to find package

Add a package (leave blank to skip): matplotlib
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search?q=matplotlib HTTP/11" 301 218
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search/?q=matplotlib HTTP/11" 200 24053
Unable to find package

Add a package (leave blank to skip): scipy
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search?q=scipy HTTP/11" 301 213
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search/?q=scipy HTTP/11" 200 23932
Unable to find package

Add a package (leave blank to skip):

Would you like to define your development dependencies interactively? (yes/no) [yes]
Package to add or search for (leave blank to skip): pytest
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search?q=pytest HTTP/11" 301 214
[urllib3:urllib3.connectionpool] https://pypi.org:443 "GET /search/?q=pytest HTTP/11" 200 23843
Unable to find package

Add a package (leave blank to skip):

Generated file

[tool.poetry]
name = "test"
version = "0.1.0"
description = ""
authors = ["peachy <[email protected]>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"


Do you confirm generation? (yes/no) [yes]

[hakuna-max]

I have verified the connectivity to https://pypi.org/simple/ using the following command:
curl -I https://pypi.org/simple/

The response confirms that the endpoint is accessible, as shown below:

HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 30348005
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: Content-Type, If-Match, If-Modified-Since, If-None-Match, If-Unmodified-Since
Access-Control-Allow-Methods: GET
Access-Control-Max-Age: 86400
Access-Control-Expose-Headers: X-PyPI-Last-Serial
X-PyPI-Last-Serial: 26316589
Cache-Control: max-age=600, public
ETag: "4xfxvUQA5tXse+HrhPZ9MQ"
Content-Security-Policy: default-src 'none'; sandbox allow-top-navigation
Referrer-Policy: origin-when-cross-origin
Accept-Ranges: bytes
Date: Tue, 03 Dec 2024 11:47:08 GMT
X-Served-By: cache-iad-kjyo7100076-IAD, cache-hkg17923-HKG
X-Cache: HIT, HIT
X-Cache-Hits: 914156, 1009
X-Timer: S1733226428.271778,VS0,VE0
Content-Type: text/html
Vary: Accept-Encoding, Accept
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
X-Frame-Options: deny
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none
Permissions-Policy: publickey-credentials-create=(self),publickey-credentials-get=(self),accelerometer=(),ambient-light-sensor=(),autoplay=(),battery=(),camera=(),display-capture=(),document-domain=(),encrypted-media=(),execution-while-not-rendered=(),execution-while-out-of-viewport=(),fullscreen=(),gamepad=(),geolocation=(),gyroscope=(),hid=(),identity-credentials-get=(),idle-detection=(),local-fonts=(),magnetometer=(),microphone=(),midi=(),otp-credentials=(),payment=(),picture-in-picture=(),screen-wake-lock=(),serial=(),speaker-selection=(),storage-access=(),usb=(),web-share=(),xr-spatial-tracking=()

This may indicate that the issue is not related to network connectivity or access to the PyPI repository. Instead, it seems specific to how poetry init handles package search queries.

[dimbleby]

indeed the search returns no results

$ curl https://pypi.org/simple/search/?q=numpy
<!DOCTYPE html>
<html>
  <head>
    <meta name="pypi:repository-version" content="1.3">
    <title>Links for search</title>
  </head>
  <body>
    <h1>Links for search</h1>
</body>
</html>
<!--SERIAL 515503-->d

perhaps something has changed or broken at https://github.com/pypi/warehouse, I expect you'll want to ask over there

[Secrus]

PyPI removed info about the latest version from their search results, breaking our search logic.

[stefansjs]

I'm surprised this was ever supported. Pip has been unable to search for years at this point

pip search numpy
ERROR: XMLRPC request failed [code: -32500]
RuntimeError: PyPI no longer supports 'pip search' (or XML-RPC search). Please use https://pypi.org/search (via a browser) instead. See https://warehouse.pypa.io/api-reference/xml-rpc.html#deprecated-methods for more information.

Probably the best solution is to just use the index directly e.g. https://pypi.org/simple/pytest/. I doesn't let the user do any kind of fuzzy matching, but it at least lets us interactively define dependencies when we're sure we know the package name exactly.

[newbery]

I'm guessing the culprit is this?
pypi/warehouse#17204

[haratyma]

I have the same issue on Mac M1 MacOs 14.6.1. Poetry 2.0.0
Any package I can not add interactively:

Package to add or search for (leave blank to skip): tiktoken
Unable to find package

Add a package (leave blank to skip): pandas
Unable to find package

Add a package (leave blank to skip): numpy
Unable to find package

[Secrus]

@abn could you check if your recent changes to search fix this?

[abn]

I don't think my recent change would fix this. We no longer can support search queries against PyPI at the moment as far as I can tell.

I have documented it at https://python-poetry.org/docs/main/cli/#search

PyPI no longer allows for the search of packages without a browser. Please use https://pypi.org/search (via a browser) instead.

For more information please see warehouse documentation and this discussion.

Although the simple query issue is separate.

[abn]

Can you please try the fix at #10055? While it does not solve partial match searches (we cannot do much about that one), it allows for exact matches with or without constraints.

Using pipx

pipx install --suffix=@10055 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/10055/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:latest <<EOF
set -xe
python -m pip install --disable-pip-version-check --root-user-action ignore -q git+https://github.com/python-poetry/poetry.git@refs/pull/10055/head
poetry new foobar
pushd foobar
poetry search pycowsay 'pytest>=8'
EOF

+ poetry search pycowsay 'pytest>=8'
 Package  Version Source Description 
 pycowsay 0.0.0.1 PyPI               
 pycowsay 0.0.0.2 PyPI               
 pytest   8.0.0   PyPI               
 pytest   8.0.1   PyPI               
 pytest   8.0.2   PyPI               
 pytest   8.1.1   PyPI               
 pytest   8.1.2   PyPI               
 pytest   8.2.0   PyPI               
 pytest   8.2.1   PyPI               
 pytest   8.2.2   PyPI               
 pytest   8.3.0   PyPI               
 pytest   8.3.1   PyPI               
 pytest   8.3.2   PyPI               
 pytest   8.3.3   PyPI               
 pytest   8.3.4   PyPI 

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.