Frequent intermittent connection failures

[maroux]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Debian GNU/Linux 10 (buster)
• Poetry version: 1.1.3
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/maroux/5bfc38375acd0e1c1a710fc0e62b8e93

Issue

Environment: Docker debian slim buster image running in Google Cloud Platform Cloud Build
Installation: poetry install -vvv --no-root --no-dev
Outcome: Fails with networking error intermittently, but frequently enough that upgrade to poetry v1.1 is blocked on solving this issue.
Notes: Older version (v1.0.9) works fine but is slow because of old installer.
Verbose logs:

16:05:17 Step #2 - "build": Skipping virtualenv creation, as specified in config file.
16:05:17 Step #2 - "build": Installing dependencies from lock file
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build": Finding the necessary packages for the current system
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build": Package operations: 17 installs, 1 update, 0 removals, 11 skipped
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ? Installing certifi (2020.6.20): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing chardet (3.0.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing idna (2.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing markupsafe (1.1.1)
16:05:17 Step #2 - "build":   ? Installing pyparsing (2.4.7): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing pytz (2020.1): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing six (1.15.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing urllib3 (1.25.10): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing alabaster (0.7.12)
16:05:17 Step #2 - "build":   ? Installing babel (2.8.0)
16:05:17 Step #2 - "build":   ? Updating colorama (0.4.4 -> 0.4.3): Skipped for the following reason: Not needed for the current environment
16:05:17 Step #2 - "build":   ? Installing docutils (0.16): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing imagesize (1.2.0)
16:05:17 Step #2 - "build":   ? Installing jinja2 (2.11.2)
16:05:17 Step #2 - "build":   ? Installing packaging (20.4): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Updating pygments (2.7.1 -> 2.6.1)
16:05:17 Step #2 - "build":   ? Installing requests (2.24.0): Skipped for the following reason: Already installed
16:05:17 Step #2 - "build":   ? Installing snowballstemmer (2.0.0)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-applehelp (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-devhelp (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-htmlhelp (1.0.2)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-jsmath (1.0.1)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-qthelp (1.0.2)
16:05:17 Step #2 - "build":   ? Installing sphinxcontrib-serializinghtml (1.1.3)
16:05:17 Step #2 - "build":   ? Installing commonmark (0.9.1)
16:05:17 Step #2 - "build":   ? Installing sphinx (3.1.2)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build":   ? Installing recommonmark (0.5.0)
16:05:17 Step #2 - "build":   ? Installing sphinx-autodoc-typehints (1.11.0)
16:05:17 Step #2 - "build":   ? Installing sphinx-rtd-theme (0.4.3)
16:05:17 Step #2 - "build": Retrying HTTP request in 0.5 seconds.
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   Stack trace:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   13  ~/.poetry/lib/poetry/installation/executor.py:199 in _execute_operation
16:05:17 Step #2 - "build":        197? 
16:05:17 Step #2 - "build":        198?             try:
16:05:17 Step #2 - "build":      ? 199?                 result = self._do_execute_operation(operation)
16:05:17 Step #2 - "build":        200?             except EnvCommandError as e:
16:05:17 Step #2 - "build":        201?                 if e.e.returncode == -2:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   12  ~/.poetry/lib/poetry/installation/executor.py:273 in _do_execute_operation
16:05:17 Step #2 - "build":        271?             return 0
16:05:17 Step #2 - "build":        272? 
16:05:17 Step #2 - "build":      ? 273?         result = getattr(self, "_execute_{}".format(method))(operation)
16:05:17 Step #2 - "build":        274? 
16:05:17 Step #2 - "build":        275?         if result != 0:
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   11  ~/.poetry/lib/poetry/installation/executor.py:408 in _execute_install
16:05:17 Step #2 - "build":        406? 
16:05:17 Step #2 - "build":        407?     def _execute_install(self, operation):  # type: (Install) -> None
16:05:17 Step #2 - "build":      ? 408?         return self._install(operation)
16:05:17 Step #2 - "build":        409? 
16:05:17 Step #2 - "build":        410?     def _execute_update(self, operation):  # type: (Update) -> None
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   10  ~/.poetry/lib/poetry/installation/executor.py:434 in _install
16:05:17 Step #2 - "build":        432?             archive = self._download_link(operation, Link(package.source_url))
16:05:17 Step #2 - "build":        433?         else:
16:05:17 Step #2 - "build":      ? 434?             archive = self._download(operation)
16:05:17 Step #2 - "build":        435? 
16:05:17 Step #2 - "build":        436?         operation_message = self.get_operation_message(operation)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    9  ~/.poetry/lib/poetry/installation/executor.py:575 in _download
16:05:17 Step #2 - "build":        573? 
16:05:17 Step #2 - "build":        574?     def _download(self, operation):  # type: (Operation) -> Path
16:05:17 Step #2 - "build":      ? 575?         link = self._chooser.choose_for(operation.package)
16:05:17 Step #2 - "build":        576? 
16:05:17 Step #2 - "build":        577?         return self._download_link(operation, link)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    8  ~/.poetry/lib/poetry/installation/chooser.py:60 in choose_for
16:05:17 Step #2 - "build":         58?         """
16:05:17 Step #2 - "build":         59?         links = []
16:05:17 Step #2 - "build":      ?  60?         for link in self._get_links(package):
16:05:17 Step #2 - "build":         61?             if link.is_wheel and not Wheel(link.filename).is_supported_by_environment(
16:05:17 Step #2 - "build":         62?                 self._env
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    7  ~/.poetry/lib/poetry/installation/chooser.py:94 in _get_links
16:05:17 Step #2 - "build":         92?             repository = self._pool.repository(package.source_reference)
16:05:17 Step #2 - "build":         93? 
16:05:17 Step #2 - "build":      ?  94?         links = repository.find_links_for_package(package)
16:05:17 Step #2 - "build":         95? 
16:05:17 Step #2 - "build":         96?         hashes = [f["hash"] for f in package.files]
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    6  ~/.poetry/lib/poetry/repositories/pypi_repository.py:245 in find_links_for_package
16:05:17 Step #2 - "build":        243? 
16:05:17 Step #2 - "build":        244?     def find_links_for_package(self, package):
16:05:17 Step #2 - "build":      ? 245?         json_data = self._get("pypi/{}/{}/json".format(package.name, package.version))
16:05:17 Step #2 - "build":        246?         if json_data is None:
16:05:17 Step #2 - "build":        247?             return []
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    5  ~/.poetry/lib/poetry/repositories/pypi_repository.py:319 in _get
16:05:17 Step #2 - "build":        317?     def _get(self, endpoint):  # type: (str) -> Union[dict, None]
16:05:17 Step #2 - "build":        318?         try:
16:05:17 Step #2 - "build":      ? 319?             json_response = self.session.get(self._base_url + endpoint)
16:05:17 Step #2 - "build":        320?         except requests.exceptions.TooManyRedirects:
16:05:17 Step #2 - "build":        321?             # Cache control redirect loop.
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    4  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:543 in get
16:05:17 Step #2 - "build":        541? 
16:05:17 Step #2 - "build":        542?         kwargs.setdefault('allow_redirects', True)
16:05:17 Step #2 - "build":      ? 543?         return self.request('GET', url, **kwargs)
16:05:17 Step #2 - "build":        544? 
16:05:17 Step #2 - "build":        545?     def options(self, url, **kwargs):
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    3  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:530 in request
16:05:17 Step #2 - "build":        528?         }
16:05:17 Step #2 - "build":        529?         send_kwargs.update(settings)
16:05:17 Step #2 - "build":      ? 530?         resp = self.send(prep, **send_kwargs)
16:05:17 Step #2 - "build":        531? 
16:05:17 Step #2 - "build":        532?         return resp
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    2  ~/.poetry/lib/poetry/_vendor/py3.7/requests/sessions.py:643 in send
16:05:17 Step #2 - "build":        641? 
16:05:17 Step #2 - "build":        642?         # Send the request
16:05:17 Step #2 - "build":      ? 643?         r = adapter.send(request, **kwargs)
16:05:17 Step #2 - "build":        644? 
16:05:17 Step #2 - "build":        645?         # Total elapsed time of the request (approximately)
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":    1  ~/.poetry/lib/poetry/_vendor/py3.7/cachecontrol/adapter.py:53 in send
16:05:17 Step #2 - "build":         51?             request.headers.update(self.controller.conditional_headers(request))
16:05:17 Step #2 - "build":         52? 
16:05:17 Step #2 - "build":      ?  53?         resp = super(CacheControlAdapter, self).send(request, **kw)
16:05:17 Step #2 - "build":         54? 
16:05:17 Step #2 - "build":         55?         return resp
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ConnectionError
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
16:05:17 Step #2 - "build": 
16:05:17 Step #2 - "build":   at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
16:05:17 Step #2 - "build":       494?                     low_conn.close()
16:05:17 Step #2 - "build":       495?                     raise
16:05:17 Step #2 - "build":       496? 
16:05:17 Step #2 - "build":       497?         except (ProtocolError, socket.error) as err:
16:05:17 Step #2 - "build":     ? 498?             raise ConnectionError(err, request=request)
16:05:17 Step #2 - "build":       499? 
16:05:17 Step #2 - "build":       500?         except MaxRetryError as e:
16:05:17 Step #2 - "build":       501?             if isinstance(e.reason, ConnectTimeoutError):
16:05:17 Step #2 - "build":       502?                 # TODO: Remove this in 3.0.0: see #2811
16:05:17 Step #2 - "build": 

[winpat]

I can confirm the same issue with Poetry 1.1.2. Every other install fails with a ConnectionError.

[abn]

Hmm, we saw this intermittently in our own CI a while ago, but was restricted to Cirrus CI FreeBSD instances (which runs on Google Cloud).

We might be able to do a bug fix to try recreate the session, but I suspect the issue might be at the process/scoket level.

@winpat what environment are you seeing your errors in?

I am really inclined to move http logic to httpx in the next minor to handle concurrent scenarios a bit better.

[winpat]

@winpat what environment are you seeing your errors in?

I observed the problem in various GitLab CI pipelines (GitLab.com, not self hosted). If I remember correctly then GitLab also uses GCP.

[dariobig]

We are having the same issue with our CircleCI builds (poetry 1.1.2, VM with ubuntu-1604:202004-01 image and python 3.7)

...
#21 423.2 
#21 423.2   ConnectionError
#21 423.2 
#21 423.2   ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
#21 423.2 
#21 423.2   at ~/.poetry/lib/poetry/_vendor/py3.7/requests/adapters.py:498 in send
#21 423.4       494│                     low_conn.close()
#21 423.4       495│                     raise
#21 423.4       496│ 
#21 423.4       497│         except (ProtocolError, socket.error) as err:
#21 423.4     → 498│             raise ConnectionError(err, request=request)
#21 423.4       499│ 
#21 423.4       500│         except MaxRetryError as e:
#21 423.4       501│             if isinstance(e.reason, ConnectTimeoutError):
#21 423.4       502│                 # TODO: Remove this in 3.0.0: see #2811
#21 423.4 
#21 426.5   • Installing pytest-timeout (1.4.2)
#21 427.7   • Installing scipy (1.5.2)
#21 428.1   • Installing threadpoolctl (2.1.0)
#21 ERROR: executor failed running [/bin/sh -c $POETRY_BINARY install]: runc did not terminate sucessfully

[abn]

I am not entirely sure if this will fix the issue, since I am assuming the real issue here is some sort of networking layer throttling being hit. If some of you can please try the fix at #3230, it would be helpful to know if session reuse across thread exeuctions was the issue here.

Using pipx

pipx install --suffix=@3230 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3230/head'

Using a container (podman | docker)

podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3230/head
poetry new foobar
pushd foobar
sed -i /pytest/d pyproject.toml
poetry add pycowsay
# commands to reproduce your build
EOF

[abn]

@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.

[winpat]

@abn Thanks for your help! I've added your fix to a couple of different projects. It will take a day or two until I can whether it's working or not.

[maroux]

@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.

I've run it four times now and it's worked every time. I believe the PR does fix this issue.

[dariobig]

Unfortunately I can't test it in our environment, I've turned off "new installer" instead and that fixed the issue for us.

…

On Tue, Oct 20, 2020 at 11:37 AM Aniruddha Maru ***@***.***> wrote: @maroux <https://github.com/maroux> @winpat <https://github.com/winpat> @dariobig <https://github.com/dariobig> can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch. I've run it four times now and it's worked every time. I believe the PR does fix this issue. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3219 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAABFE553C5KZ7MJQTQMHVTSLWVEPANCNFSM4SSUHTOA> .

[winpat]

@maroux @winpat @dariobig can you please verify if the fix above resolves the issue for your environments. If it does I can get that into the next patch.

I've run it four times now and it's worked every time. I believe the PR does fix this issue.

I also ran it a couple of times without any issues so far 👍.

[sloev]

when is this gonna be fixed? this morning: again 9 consecutive travis builds before i got thru my poetry install step. poetry version: 1.1.2

i've now recognized my posts here and on the other issues were unnecesarily blunt and had a tone i dont stand for. So i apologize for my behavior and wish no harm was done. I was under stress and this was an accidental side-effect.
Hope you all have a nice day and i will see if i can find a way to contribute constructively to this great project going forward.
...johannes

[abn]

@sloev as I mentioned on the pull request (#3230 (comment)), this is an open source community project, not a product you have a paid support for. This is maintained by a small group of volunteers who spend their spare time on this. You demanding issues be fixed and spamming issue threads is not going to make anything go any faster. While we can appreciate that this is causing issues with your workflow, we have mentioned in the release post how to disable the experimental installer to unblock yourself. Please keep the conversation constructive and civil.

[abn]

@DustinMoriarty yeah; we noticed that. This should go away once 1.1.4 is out. Hopefully the fix in #3230 is sufficient.

[abn]

This was resolved with 1.1.4.

[maroux]

uh-oh just saw this now on 1.1.4:

13:08:47 Step #2 - "build":   ChunkedEncodingError
13:08:47 Step #2 - "build": 
13:08:47 Step #2 - "build":   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
13:08:47 Step #2 - "build": 
13:08:47 Step #2 - "build":   at ~/.poetry/lib/poetry/_vendor/py3.7/requests/models.py:754 in generate
13:08:47 Step #2 - "build":       750?                 try:
13:08:47 Step #2 - "build":       751?                     for chunk in self.raw.stream(chunk_size, decode_content=True):
13:08:47 Step #2 - "build":       752?                         yield chunk
13:08:47 Step #2 - "build":       753?                 except ProtocolError as e:
13:08:47 Step #2 - "build":     ? 754?                     raise ChunkedEncodingError(e)
13:08:47 Step #2 - "build":       755?                 except DecodeError as e:
13:08:47 Step #2 - "build":       756?                     raise ContentDecodingError(e)
13:08:47 Step #2 - "build":       757?                 except ReadTimeoutError as e:
13:08:47 Step #2 - "build":       758?                     raise ConnectionError(e)
13:08:47 Step #2 - "build": 
13:08:47 Step #2 - "build":   ? Installing snowballstemmer (2.0.0)
13:08:47 Step #2 - "build":   ? Installing sphinxcontrib-applehelp (1.0.1)
13:09:01 ERROR: (gcloud.builds.submit) build 17e8547a-8452-4010-a146-81c7f21a4309 completed with status "FAILURE"

might be unrelated?

[abn]

@maroux I suspect this might be different as this is connection being broken during a download.

[Korijn]

We're still seeing exactly this problem on our CI builds (azure pipelines) with poetry-1.1.4, our docker base image is python:3.6-slim-buster just like OP:

#18 13.66   • Installing black (19.10b0)
#18 13.83 
#18 13.83   ConnectionError
#18 13.83 
#18 13.83   ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
#18 13.83 
#18 13.83   at /usr/local/lib/python3.6/site-packages/requests/adapters.py:498 in send
#18 13.87       494│                     low_conn.close()
#18 13.87       495│                     raise
#18 13.87       496│ 
#18 13.87       497│         except (ProtocolError, socket.error) as err:
#18 13.87     → 498│             raise ConnectionError(err, request=request)
#18 13.87       499│ 
#18 13.87       500│         except MaxRetryError as e:
#18 13.87       501│             if isinstance(e.reason, ConnectTimeoutError):
#18 13.87       502│                 # TODO: Remove this in 3.0.0: see #2811

Since there's a pretty big list of packages to install, this happens almost every build. We're pinning to poetry < 1.1 for now as a workaround.

[SimonBiggs]

Since there's a pretty big list of packages to install, this happens almost every build. We're pinning to poetry < 1.1 for now as a workaround.

As per https://python-poetry.org/blog/announcing-poetry-1-1-0.html#brand-new-installer adding something like the following to your CI builds might work:

https://github.com/pymedphys/pymedphys/blob/765f6229105f221c3dd75ff9927e62dfde1b3972/.github/workflows/test-and-deploy.yml#L154-L156

[prnvkbr]

Unfortunately, still experiencing this on poetry-1.1.5. Has anyone else experienced this issue lately?

...
#22 177.1 Package operations: 47 installs, 0 updates, 0 removals
#22 177.1 
#22 177.1   • Installing numpy (1.20.1)
#22 177.1   • Installing typing-extensions (3.7.4.3)
#22 184.8   • Installing markupsafe (1.1.1)
#22 184.8   • Installing pillow (8.1.2)
#22 184.8   • Installing six (1.15.0)
#22 184.8   • Installing smmap (3.0.5)
#22 184.8   • Installing torch (1.7.1)
#22 195.9 
#22 195.9   ChunkedEncodingError
#22 195.9 
#22 195.9   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
#22 195.9 
#22 195.9   at /opt/conda/lib/python3.8/site-packages/requests/models.py:754 in generate
#22 196.1       750│                 try:
#22 196.1       751│                     for chunk in self.raw.stream(chunk_size, decode_content=True):
#22 196.1       752│                         yield chunk
#22 196.1       753│                 except ProtocolError as e:
#22 196.1     → 754│                     raise ChunkedEncodingError(e)
#22 196.1       755│                 except DecodeError as e:
#22 196.1       756│                     raise ContentDecodingError(e)
#22 196.1       757│                 except ReadTimeoutError as e:
#22 196.1       758│                     raise ConnectionError(e)
#22 196.1 
------
Dockerfile:21
--------------------
  19 |     RUN pip install --upgrade pip && pip --no-cache-dir install poetry
  20 |     COPY ./pyproject.toml .
  21 | >>> RUN poetry install --verbose

Base Docker image: continuumio/miniconda3.

Any workarounds?

[abn]

@danielgafni can you open a new issue for this? The workers need to be configurable. We definitely are a bit greedy, but then again I suspct what we need to connection pooling for repository sessions so installations can continue faster if all dependencies are already cached for example.

poetry/poetry/installation/executor.py

Line 75 in c6b19b3

self._max_workers = os.cpu_count() + 4

[cposada23]

Hello I keep getting this

 ('Connection aborted.', TimeoutError(10060, 'A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond', None, 10060, None))

  at ~\AppData\Roaming\pypoetry\venv\lib\site-packages\requests\adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811

When I run poetry Install, have anyone find a solution ?

[danielgafni]

hey @abn ! I'm sorry, I totally forgot about this thread.

I've found a PR that implements exactly this, it looks like it's almost ready.

[rbudnar]

We're experiencing this issue as well very consistently in our repositories that include a link to our private Azure package feed. We've tried the solution posted above to no avail. Note this is on poetry version 1.1.12.
Error:

Updating dependencies
Resolving dependencies... (295.3s)

  ConnectionError

  ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

  at ~/.poetry/lib/poetry/_vendor/py3.9/requests/adapters.py:498 in send
      494│                     low_conn.close()
      495│                     raise
      496│ 
      497│         except (ProtocolError, socket.error) as err:
    → 498│             raise ConnectionError(err, request=request)
      499│ 
      500│         except MaxRetryError as e:
      501│             if isinstance(e.reason, ConnectTimeoutError):
      502│                 # TODO: Remove this in 3.0.0: see #2811

We've noticed that when we remove our private packages and package feed from our list, the problem goes away immediately; also dependency resolution is much much faster (in this case, from 100-300 seconds with failure, down to ~20 seconds with success).

[rbudnar]

@abn Do you have any insights on how to resolve #3219 (comment) ?

[mbelang]

I have the same issue as @rbudnar with 1.1.12 with Gemfury. The failures are really uneven though sometimes it fails right away, sometimes it takes 30 secs, sometimes more.

I just manage to do an install after 15-20 retries :(

[gshpychka]

@abn I'm also still running into this, even with installer.parallel false, when running on AWS CodeBuild.

[abn]

@gshpychka @mbelang can you try with poetry@master maybe? If that does not work, maybe raise a new issue?

[johnthagen]

This happened for me in CI with a build machine with 32 cores.

• GitLab CI
• Poetry 1.1.13
• Docker executor running python:3.9

  ConnectionError
  HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /pypi/tomli/2.0.1/json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fb8b979fdf0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
  at /usr/local/lib/python3.9/site-packages/requests/adapters.py:519 in send
      515│             if isinstance(e.reason, _SSLError):
      516│                 # This branch is for urllib3 v1.22 and later.
      517│                 raise SSLError(e, request=request)
      518│ 
    → 519│             raise ConnectionError(e, request=request)
      520│ 
      521│         except ClosedPoolError as e:
      522│             raise ConnectionError(e, request=request)
      523│ 

[roscioli]

+1 -- happening on poetry 1.1.13 in a docker container :(

[taranlu-houzz]

I am also still experiencing this. I have not had time to investigate further, but I am wondering if it might have to do with using a private pypi repo as an additional source? I don't seem to run into this issue when I am working on packages that don't have the tool.poetry.source entry. Seems like maybe it could be a combination of the private pypi server, VPN settings, and potentially underpowered hardware for the pypi server, since the issues are due to timeouts...

[roscioli]

My hunch is that it also may have to do with internet speed. I'm succeeding at 330 Mbps but my colleague is failing at 27 Mbps. I think perhaps the retries need to be more forgiving?

[estyxx]

I'm experiencing this as well with Poetry 1.1.13 in a docker container, and I'm don't have a private repo...

[invokermain]

Also seeing this frequently on our Gitlab CI/CD pipelines (running in container) with a private repo as a secondary source

[danielgafni]

Just fyi if you haven't tried it yet: setting poetry config installer.parallel false fixes the issue for me in 100% of the cases. We are running a lot of CI with private repos and this issue never occurs for us (with the correct setting). It may occur if installing Poetry projects on machines with a lot of CPUs (without the setting).

[invokermain]

@danielgafni I've added that to our pipelines today, shame that it makes the pipelines dramatically slower (and the first pipeline we ran after it failed with the error). But hopefully it will help a bit.

Out of interest are your private repos set as 'secondary = True' or not?

[danielgafni]

@invokermain yes, it's set to true

[invokermain]

Yeah a few days later and we are still getting this issue very regularly, even with poetry config installer.parallel false. I think one of the underlying issues that means it's more frequent in CI/CD is the fact that you are less likely to have caching enabled. I.e. for us there is no caching and therefore the installs are much more spammy (especially if running multiple jobs, e.g. test/quality/security, in parallel).

But having some sort of exponential backoff on the poetry side would be a good fix for this no?

[invokermain]

Hmmm looks like this might be resolved on 1.2, so 🤞 for a release soon!

[illyakaynov]

Still experiencing this problem on 1.2.2

[mohghaderi]

Still experiencing this problem on 1.4.x
poetry config installer.parallel false works almost always

poetry config installer.max-workers 4 still fails most of the time
poetry config installer.max-workers 2 works

I think there should be a problem with the way parallel works.

[omerbe-nym]

Still experiencing this problem on 1.5.1

ConnectionResetError
 
   [Errno 104] Connection reset by peer
 
   at /usr/lib/python3.7/ssl.py:929 in read
        925│         if self._sslobj is None:
        926│             raise ValueError("Read on closed or unwrapped SSL socket.")
        927│         try:
        928│             if buffer is not None:
     →  929│                 return self._sslobj.read(len, buffer)
        930│             else:
        931│                 return self._sslobj.read(len)
        932│         except SSLError as x:
        933│             if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:
 
 The following error occurred when trying to handle this error:
 
 
   ProtocolError
 
   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
 
   at /opt/poetry/venv/lib/python3.7/site-packages/urllib3/response.py:461 in _error_catcher
       457│                 raise ReadTimeoutError(self._pool, None, "Read timed out.")
       458│ 
       459│             except (HTTPException, SocketError) as e:
       460│                 # This includes IncompleteRead.
     → 461│                 raise ProtocolError("Connection broken: %r" % e, e)
       462│ 
       463│             # If no exception is thrown, we should avoid cleaning up
       464│             # unnecessarily.
       465│             clean_exit = True
 
 The following error occurred when trying to handle this error:
 
 
   ChunkedEncodingError
 
   ("Connection broken: ConnectionResetError(104, 'Connection reset by peer')", ConnectionResetError(104, 'Connection reset by peer'))
 
   at /opt/poetry/venv/lib/python3.7/site-packages/requests/models.py:818 in generate
        814│             if hasattr(self.raw, "stream"):
        815│                 try:
        816│                     yield from self.raw.stream(chunk_size, decode_content=True)
        817│                 except ProtocolError as e:
     →  818│                     raise ChunkedEncodingError(e)
        819│                 except DecodeError as e:
        820│                     raise ContentDecodingError(e)
        821│                 except ReadTimeoutError as e:
        822│                     raise ConnectionError(e)

[ezorita]

Same behavior with poetry 1.5.1, intermittent failures when installing in docker containers or while building images. @abn would you consider reopening this issue to investigate? Thanks!

[coltonb-arcadia]

@omerbe-nym @ezorita

If you're still experiencing this, it's possible that your docker container is reporting a ton of cores (whether that is true or not, see below) when os.cpu_count() is called by poetry. See #3219 (comment)

The max number of workers has no upper bound by default, meaning hundreds of connections might be attempted and this error occurs. Setting max-workers to a reasonable value, e.g. somewhere between 5 - 8, will likely resolve this.

POETRY_INSTALLER_MAX_WORKERS=5 poetry install

We encountered this on CircleCI - CircleCI's Python convenience image reports 36 cores regardless of which resource class you're using.

[gehirndienst]

I confirm the same issue with poetry 1.6.1, the only thing that brings stability (in a turtle-speed way though) is poetry config installer.parallel false

we noticed however that this problem happens only to customers with nvidia-packages (cuda, cublas, cudnn, etc.) in the installation chain and only by building a docker image on Windows.

[maronuu]

I also confirmed the same issue with poetry 1.6.1, even when poetry config installer.parallel false is set.
I only encountered the error while building a docker image from Dockerfile with Linux OS. Docker is running on MacOS. In my case, various package (non-deterministic) including nvidia-packages failed.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.