Update: add DNS caching to request manager #4447
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
**Summary** Fixes #746. Unfortunately neither Node, nor many systems come with built-in DNS caching so the many parallel requests that Yarn makes sometimes overwhelm the DNS servers, and most of the time, for the very same domain(s). Even worse, we pay the DNS look up cost for each request, which is quite sad at best. This patch introduces the `dnscache` module which intercepts all DNS look ups and answers them from an in-memory cache when possible. This applies to the built-in `http` and `https` modules, used by `request`. **Test plan** Exiting tests should pass, and hopefully be faster. Total number of DNS look ups should decrease dramatically.
kaylie-alexa
approved these changes
Sep 14, 2017
|
This change will increase the build size from 9.52 MB to 9.69 MB, an increase of 176.98 KB (2%)
|
|
70 KB increase seems like a lot, but I guess it's worth it 😛 I'm surprised the operating system doesn't do DNS caching? Does this mean that every app needs to implement its own cache? 😢 |
|
Good catch.
Is it JS only btw, does it bundle well?
…On Wed, Sep 13, 2017 at 9:36 PM Daniel Lo Nigro ***@***.***> wrote:
70 KB increase seems like a lot, but I guess it's worth it 😛
I'm surprised the operating system doesn't do DNS caching? Does this mean
that every app needs to implement its own cache? 😢
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
<#4447 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ACBdWOlvO1tTv_txOKelSsoi2-y_yobsks5siK1CgaJpZM4PW0Ju>
.
|
|
@bestander see #4447 (comment) I think it is acceptable since it is a 2% increase.
Some do some doesn't and apparently even if they do, they are not great. I know Chrome comes with its own DNS cache and resolution system for instance. Windows is supposed have the cache but those Docker instances sure don't and I'm fairly certain that leaving the app for the exact same domain thousands of times (even if you have the OS cache) is not great. |
|
any numbers on which scale this changes improves performance? |
joaolucasl
pushed a commit
to joaolucasl/yarn
that referenced
this pull request
Oct 27, 2017
**Summary** Fixes yarnpkg#746. Unfortunately, neither Node, nor many systems come with built-in DNS caching so the many parallel requests that Yarn makes sometimes overwhelm the DNS servers, and most of the time, for the very same domain(s). Even worse, we pay the DNS look up cost for each request, which is quite sad at best. This patch introduces the `dnscache` module which intercepts all DNS look ups and answers them from an in-memory cache when possible. This applies to the built-in `http` and `https` modules, used by `request`. **Test plan** Existing tests should pass, and hopefully be faster. Total number of DNS look ups should decrease dramatically.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #746. Unfortunately, neither Node, nor many systems come with
built-in DNS caching so the many parallel requests that Yarn makes
sometimes overwhelm the DNS servers, and most of the time, for the
very same domain(s). Even worse, we pay the DNS look up cost for
each request, which is quite sad at best. This patch introduces
the
dnscachemodule which intercepts all DNS look ups and answersthem from an in-memory cache when possible. This applies to the
built-in
httpandhttpsmodules, used byrequest.Test plan
Existing tests should pass, and hopefully be faster. Total number of
DNS look ups should decrease dramatically.