Fetching from GitHub raises a Connection reset by peer error

[albertfdp]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

When running yarn on package.json that contains many repositories pointing to GitHub, this will cause yarn to fail on the fetch step.

If the current behavior is a bug, please provide the steps to reproduce.

I have created a example repository with a package.json that I can reproduce the error https://github.com/albertfdp/yarn-github-repos/blob/master/package.json.

1. Checkout https://github.com/albertfdp/yarn-github-repos
2. Run yarn

On the fetch step, it raises an error like this (note, it's not always the same dependency that fails):

yarn install v0.16.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error Command failed.
Exit code: 128
Command: git
Arguments: ls-remote --tags --heads ssh://[email protected]/ReactTraining/react-router.git
Directory: /PATH/yarn-github-repos
Output:
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

(in a different run):

yarn install v0.16.1
info No lockfile found.
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
error Command failed.
Exit code: 1
Command: git
Arguments: pull
Directory: /PATH/.yarn-cache/.tmp/00ce03dbdd7a458a1f39defd7ff9dcf5
Output:
ssh_exchange_identification: read: Connection reset by peer
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.

The real use-case I have is for private GitHub repos, the repos in this example are public and I should use the npm registry, but it's only for helping to reproduce the bug.

What is the expected behavior?

That fetching packages from GitHub works just as with the npm-cli.

Please mention your node.js, yarn and operating system version.

• node v5.6.0
• yarn v0.16.1
• macOS 10.11.6

[chase]

Works correctly with Node v6.6, yarn v0.16.1, and Linux 4.6.7-1.

I'm using git version 2.10.0, and git ls-remote --tags --heads ssh://[email protected]/ReactTraining/react-router.git also works correctly.

What version of git are you using @albertfdp?

[albertfdp]

It seems like upgrading to the last version of git (2.10.1) solves this issue and I can't reproduce it anymore.

Thanks anyway!

[albertfdp]

I'll reopen this issue as this is still a problem, although I have been able to identify the root cause: The problem is that using multiple concurrent ssh connections to GitHub from the same IP means that you are flagged as a possible DDoS and thus GitHub throws you with the Connection reset by peer. ssh connections are much more "expensive" than HTTP, and probably GitHub has a lower rate limit.

I have consistently been able to reproduce this with the reproduction steps above while on my organization network, but it works when using a mobile network or my home network, since obviously those are slower connections that cannot handle so many concurrent requests and will prevent this issue. This is the reason why I mistakenly thought it was a git issue on the previous comment, as I was using then my home connection. Also, after you receive this error, you are flagged and during a period of time, npm install will also fail with the same error, which reinforces that this is concurrency issue.

I think it would be useful to be able to control the concurrency, especially because it's only an issue when you need to warm the cache, on later yarn install this problem does not arise. I am happy to submit a PR surfacing the NETWORK_CONCURRENCY as a CLI option https://github.com/yarnpkg/yarn/blob/master/src/constants.js#L27.