https: fix memory leak with https.request() #8647

imyller · 2016-09-18T12:40:14Z

Checklist

make -j4 test (UNIX), or vcbuild test nosign (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

https

Description of change

If calling https.request() with options.headers.host defined and options.servername undefined, https.Agent.createSocket mutates connection options after https.Agent.addRequest has created empty socket pool array with mismatching connection name.

This results in two socket pool arrays being created and only the last one gets eventually deleted by removeSocket - effectively causing a memory leak.

This commit fixes the leak by making sure that addRequest does the same modifications to options object as the createSocket.

createSocket is intentionally left unmodified to prevent userland regressions.

Test case included.

Fixes: #6687

/cc @nodejs/http @nodejs/crypto

imyller · 2016-09-18T12:43:56Z

CI: https://ci.nodejs.org/job/node-test-pull-request/4106/

mscdex · 2016-09-18T12:47:58Z

lib/_http_agent.js

+    options.servername = options.host;
+    const hostHeader = req.getHeader('host');
+    if (hostHeader) {
+      options.servername = hostHeader.replace(/:.*$/, '');


I'd prefer just using hostHeader.slice() and hostHeader.indexOf(), but if others are not comfortable with that, at the very least the regex should be /:[\s\S]*$/ to match any character (including newlines).

I left the hostHeader cleanup intentionally unmodified to match the original feature implemented PR #1110

I did not want to fix two unrelated issues in single PR dedicated to memory leak.

cjihrig · 2016-09-18T12:53:31Z

test/parallel/test-https-agent-sockets-leak.js

+    ca: options.ca,
+    agent: agent
+  }, common.mustCall((res) => {
+


Can you remove the extra blank lines at lines 37, 44, 48, 54, and 56.

Extra blank lines removed

indutny · 2016-09-18T13:00:40Z

LGTM

cjihrig

LGTM

JacksonTian · 2016-09-18T13:18:41Z

LGTM

imyller · 2016-09-18T14:14:23Z

FreeBSD CI failed due to force pushed commit

New CI run: https://ci.nodejs.org/job/node-test-pull-request/4108/

imyller · 2016-09-18T14:19:25Z

I'm going to add lts-agenda label because this leak affects current LTS release too.
Feel free to remove if fixing this in LTS is not relevant.

imyller · 2016-09-19T16:36:05Z

CI before landing: https://ci.nodejs.org/job/node-test-pull-request/4127/

jasnell

LGTM

lpinca · 2016-09-20T19:47:41Z

test/parallel/test-https-agent-sockets-leak.js

+      // To verify that no leaks occur, check that no entries
+      // exist in agent.sockets pool after both request and socket
+      // has been closed.
+


I think this line should also be removed? #8647 (comment)

Thanks. Removed extra line.

If calling `https.request()` with `options.headers.host` defined and `options.servername` undefined, `https.Agent.createSocket` mutates connection `options` after `https.Agent.addRequest` has created empty socket pool array with mismatching connection name. This results in two socket pool arrays being created and only the last one gets eventually deleted by `removeSocket` - causing a memory leak. This commit fixes the leak by making sure that `addRequest` does the same modifications to `options` object as the `createSocket`. `createSocket` is intentionally left unmodified to prevent userland regressions. Test case included. Fixes: nodejs#6687

jasnell · 2016-09-20T23:10:36Z

Trying CI again due to the AIX error: https://ci.nodejs.org/job/node-test-pull-request/4179/
Seems unrelated but just to be safe...

imyller · 2016-09-20T23:16:20Z

If CI is green, I'll land this.

imyller · 2016-09-21T00:13:44Z

I'll start landing this:

~~Three~~ Four LGTMs
No objections
Requested modifications have been made
CI tests passed (only unrelated CI failures)

If calling `https.request()` with `options.headers.host` defined and `options.servername` undefined, `https.Agent.createSocket` mutates connection `options` after `https.Agent.addRequest` has created empty socket pool array with mismatching connection name. This results in two socket pool arrays being created and only the last one gets eventually deleted by `removeSocket` - causing a memory leak. This commit fixes the leak by making sure that `addRequest` does the same modifications to `options` object as the `createSocket`. `createSocket` is intentionally left unmodified to prevent userland regressions. Test case included. PR-URL: nodejs#8647 Fixes: nodejs#6687 Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jackson Tian <[email protected]> Reviewed-By: James M Snell <[email protected]>

imyller · 2016-09-21T00:23:41Z

landed in db5a879

MylesBorins · 2016-09-30T21:10:42Z

@imyller I'm removing the agenda label for right now as there does not appear to be anything controversial about this change. After it has lived in a release for at least two weeks we can backport

If calling `https.request()` with `options.headers.host` defined and `options.servername` undefined, `https.Agent.createSocket` mutates connection `options` after `https.Agent.addRequest` has created empty socket pool array with mismatching connection name. This results in two socket pool arrays being created and only the last one gets eventually deleted by `removeSocket` - causing a memory leak. This commit fixes the leak by making sure that `addRequest` does the same modifications to `options` object as the `createSocket`. `createSocket` is intentionally left unmodified to prevent userland regressions. Test case included. PR-URL: #8647 Fixes: #6687 Reviewed-By: Fedor Indutny <[email protected]> Reviewed-By: Colin Ihrig <[email protected]> Reviewed-By: Jackson Tian <[email protected]> Reviewed-By: James M Snell <[email protected]>

nodejs-github-bot added the http Issues or PRs related to the http subsystem. label Sep 18, 2016

imyller added memory Issues and PRs related to the memory management or memory footprint. https Issues or PRs related to the https subsystem. labels Sep 18, 2016

imyller force-pushed the https-request-leak branch from 43ac439 to 0891876 Compare September 18, 2016 12:43

mscdex reviewed Sep 18, 2016

View reviewed changes

cjihrig reviewed Sep 18, 2016

View reviewed changes

imyller force-pushed the https-request-leak branch from 0891876 to b6b452c Compare September 18, 2016 12:59

cjihrig approved these changes Sep 18, 2016

View reviewed changes

indutny approved these changes Sep 18, 2016

View reviewed changes

imyller added the lts-agenda label Sep 18, 2016

Trott added the lts-watch-v4.x label Sep 18, 2016

jasnell approved these changes Sep 20, 2016

View reviewed changes

lpinca reviewed Sep 20, 2016

View reviewed changes

imyller force-pushed the https-request-leak branch from b6b452c to 4d68ae2 Compare September 20, 2016 19:51

imyller self-assigned this Sep 20, 2016

Trott force-pushed the master branch from b0df363 to c5ce7f4 Compare September 21, 2016 00:10

imyller closed this Sep 21, 2016

imyller removed their assignment Sep 21, 2016

MylesBorins removed the lts-agenda label Sep 30, 2016

imyller mentioned this pull request Oct 7, 2016

Memory leak for https.request() on ECONNRESET #8827

Closed

MylesBorins added land-on-v4.x and removed lts-watch-v4.x labels Nov 18, 2016

MylesBorins mentioned this pull request Nov 22, 2016

v4.7.0 proposal #9736

Merged

snyk-bot mentioned this pull request Jul 17, 2020

[Snyk] Security upgrade eslint from 3.19.0 to 4.0.0 adamlaska/node#38

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

https: fix memory leak with https.request() #8647

https: fix memory leak with https.request() #8647

imyller commented Sep 18, 2016 •

edited

Loading

imyller commented Sep 18, 2016

mscdex Sep 18, 2016

imyller Sep 18, 2016

cjihrig Sep 18, 2016

imyller Sep 18, 2016

imyller Sep 18, 2016

indutny commented Sep 18, 2016

cjihrig left a comment

JacksonTian commented Sep 18, 2016

imyller commented Sep 18, 2016

imyller commented Sep 18, 2016

imyller commented Sep 19, 2016

jasnell left a comment

lpinca Sep 20, 2016

imyller Sep 20, 2016

jasnell commented Sep 20, 2016

imyller commented Sep 20, 2016

imyller commented Sep 21, 2016 •

edited

Loading

imyller commented Sep 21, 2016

MylesBorins commented Sep 30, 2016

https: fix memory leak with https.request() #8647

https: fix memory leak with https.request() #8647

Conversation

imyller commented Sep 18, 2016 • edited Loading

Checklist

Affected core subsystem(s)

Description of change

imyller commented Sep 18, 2016

mscdex Sep 18, 2016

Choose a reason for hiding this comment

imyller Sep 18, 2016

Choose a reason for hiding this comment

cjihrig Sep 18, 2016

Choose a reason for hiding this comment

imyller Sep 18, 2016

Choose a reason for hiding this comment

imyller Sep 18, 2016

Choose a reason for hiding this comment

indutny commented Sep 18, 2016

cjihrig left a comment

Choose a reason for hiding this comment

JacksonTian commented Sep 18, 2016

imyller commented Sep 18, 2016

imyller commented Sep 18, 2016

imyller commented Sep 19, 2016

jasnell left a comment

Choose a reason for hiding this comment

lpinca Sep 20, 2016

Choose a reason for hiding this comment

imyller Sep 20, 2016

Choose a reason for hiding this comment

jasnell commented Sep 20, 2016

imyller commented Sep 20, 2016

imyller commented Sep 21, 2016 • edited Loading

imyller commented Sep 21, 2016

MylesBorins commented Sep 30, 2016

imyller commented Sep 18, 2016 •

edited

Loading

imyller commented Sep 21, 2016 •

edited

Loading