module: prefer async/await in https imports #41950

benjamingr · 2022-02-12T18:50:10Z

cc @bmeck @nodejs/modules

This PR refactors the fetch_module fetchWithRedirects function to use the new modern facilities for streams:

compose instead of pipe for simplified and "more easily correct" error handling on the body stream cc @ronag .
async/await over new Promise.
toArray() on the response body instead of manually.

Note this can probably be further simplified by always caching the promise (and never the value) and by moving the "memoize" functionality to a helper).

Fixes: #41950

nodejs-github-bot · 2022-02-12T18:50:14Z

Review requested:

@nodejs/modules

GeoffreyBooth · 2022-02-13T00:04:00Z

cc @JakobJingleheimer

GeoffreyBooth

Some minor style notes, I haven’t reviewed in depth.

lib/internal/modules/esm/fetch_module.js

benjamingr · 2022-02-13T10:26:33Z

@GeoffreyBooth fixed + added http 404 handling

lib/internal/modules/esm/fetch_module.js

bmeck · 2022-02-13T13:52:11Z

Note this can probably be further simplified by always caching the promise (and never the value) and by moving the "memoize" functionality to a helper).

This is actually a bit complicated since import.meta.url needs the value synchronously for the response location after all redirects so it actually relies on storing the value for now.

benjamingr · 2022-02-13T14:51:59Z

This is actually a bit complicated since import.meta.url needs the value synchronously for the response location after all redirects so it actually relies on storing the value for now.

Ah, mind if I add a comment to the code explaining that around the cacheForGET code?

bmeck · 2022-02-13T15:26:44Z

Ah, mind if I add a comment to the code explaining that around the cacheForGET code?

I'm all for comments

benjamingr · 2022-02-13T16:25:02Z

Added both comments.

JakobJingleheimer

Oow, fancy. Much cleaner than .on(…) 🙌

JakobJingleheimer · 2022-02-15T18:55:08Z

lib/internal/modules/esm/fetch_module.js

@@ -32,6 +30,9 @@ const path = require('path');
 /**
 * Only for GET requests, other requests would need new Map
 * HTTP cache semantics keep diff caches
+ * 
+ * It caches either the promise or the cahce entry since import.meta.url needs


Suggested change

* It caches either the promise or the cahce entry since import.meta.url needs

* It caches either the promise or the cache entry since import.meta.url needs

lib/internal/modules/esm/fetch_module.js

benjamingr · 2022-02-16T07:53:02Z

@bmeck @nodejs/modules this could use some reviews :)

guybedford

Can we get a 404 test though to land?

benjamingr · 2022-02-16T18:05:30Z

Added test :)

nodejs-github-bot · 2022-02-16T18:13:36Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42625/

nodejs-github-bot · 2022-02-18T11:26:22Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42649/

nodejs-github-bot · 2022-02-23T10:39:31Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42731/

nodejs-github-bot · 2022-02-23T11:20:41Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42735/

nodejs-github-bot · 2022-02-23T16:36:26Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42742/

nodejs-github-bot · 2022-02-23T18:10:07Z

CI: https://ci.nodejs.org/job/node-test-pull-request/42748/

nodejs-github-bot · 2022-02-23T20:24:29Z

Commit Queue failed

- Loading data for nodejs/node/pull/41950
✔  Done loading data for nodejs/node/pull/41950
----------------------------------- PR info ------------------------------------
Title      module: prefer async/await in https imports (#41950)
Author     Benjamin Gruenbaum  (@benjamingr)
Branch     benjamingr:use-async-await-in-fetch-module -> nodejs:master
Labels     esm, needs-ci
Commits    1
 - module: prefer async/await in https imports
Committers 1
 - Benjamin Gruenbaum 
PR-URL: https://github.com/nodejs/node/pull/41950
Fixes: https://github.com/nodejs/node/issues/41950
Reviewed-By: Guy Bedford 
Reviewed-By: James M Snell 
------------------------------ Generated metadata ------------------------------
PR-URL: https://github.com/nodejs/node/pull/41950
Fixes: https://github.com/nodejs/node/issues/41950
Reviewed-By: Guy Bedford 
Reviewed-By: James M Snell 
--------------------------------------------------------------------------------
   ⚠  Commits were pushed since the last review:
   ⚠  - module: prefer async/await in https imports
   ℹ  This PR was created on Sat, 12 Feb 2022 18:50:10 GMT
   ✔  Approvals: 2
   ✔  - Guy Bedford (@guybedford): https://github.com/nodejs/node/pull/41950#pullrequestreview-884778590
   ✔  - James M Snell (@jasnell) (TSC): https://github.com/nodejs/node/pull/41950#pullrequestreview-889949951
   ✔  Last GitHub CI successful
   ℹ  Last Full PR CI on 2022-02-23T18:10:07Z: https://ci.nodejs.org/job/node-test-pull-request/42748/
- Querying data for job/node-test-pull-request/42748/
   ✔  Last Jenkins CI successful
--------------------------------------------------------------------------------
   ✔  Aborted `git node land` session in /home/runner/work/node/node/.ncu

https://github.com/nodejs/node/actions/runs/1889467631

benjamingr · 2022-02-23T22:00:03Z

Landed in 7efef74 🎉

PR-URL: #41950 Fixes: #41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs#41950 Fixes: nodejs#41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

juanarbol · 2022-05-30T21:22:21Z

This is failing in the v16.x branch; let me work in a backport for this:

/node --inspect --experimental-network-imports --dns-result-order=ipv4first test/es-module/test-http-imports.mjs                                                                                 
Debugger listening on ws://127.0.0.1:9229/924045ba-ef66-4869-84bf-085849cbf26b                                                                                                                      
For help, see: https://nodejs.org/en/docs/inspector                                                                                                                                                 
node:internal/errors:465                                                                                                                                                                            
    ErrorCaptureStackTrace(err);                                                                                                                                                                    
    ^                                                                                                                                                                                               
                                                                                                                                                                                                    
TypeError [ERR_INVALID_ARG_TYPE]: The "list" argument must be an instance of Array. Received an instance of Buffer                                                                                  
    at concat (node:buffer:537:3)                                                                                                                                                                   
    at node:internal/modules/esm/fetch_module:165:24                                                                                                                                                
    at processTicksAndRejections (node:internal/process/task_queues:96:5)                                                                                                                           
    at async node:internal/modules/esm/fetch_module:171:7                                                                                                                                           
    at async defaultLoad (node:internal/modules/esm/load:21:14)                                                                                                                                         at async ESMLoader.load (node:internal/modules/esm/loader:407:20)                                                                                                                               
    at async ESMLoader.moduleProvider (node:internal/modules/esm/loader:326:11)                                                                                                                     
    at async link (node:internal/modules/esm/module_job:70:21) {                                                                                                                                    
  code: 'ERR_INVALID_ARG_TYPE'                                                                                                                                                                      
}

Looks like the toArray() method is returning the Buffer not the array, see: https://github.com/nodejs/node/pull/41553/files#diff-00d8d040a3fe90e3052af39cb55dc9bcac70539d5d17791449cbfe7e3b55fb90R1903-R1904

PR-URL: #41950 Fixes: #41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs/node#41950 Fixes: nodejs/node#41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

nodejs-github-bot added esm Issues and PRs related to the ECMAScript Modules implementation. needs-ci PRs that need a full CI run. labels Feb 12, 2022

benjamingr mentioned this pull request Feb 12, 2022

HTTP imports not found error #41948

Closed

GeoffreyBooth requested a review from bmeck February 13, 2022 00:03

GeoffreyBooth reviewed Feb 13, 2022

View reviewed changes

lib/internal/modules/esm/fetch_module.js Outdated Show resolved Hide resolved

lib/internal/modules/esm/fetch_module.js Outdated Show resolved Hide resolved

lib/internal/modules/esm/fetch_module.js Outdated Show resolved Hide resolved

bmeck reviewed Feb 13, 2022

View reviewed changes

lib/internal/modules/esm/fetch_module.js Show resolved Hide resolved

bmeck reviewed Feb 13, 2022

View reviewed changes

lib/internal/modules/esm/fetch_module.js Show resolved Hide resolved

bmeck reviewed Feb 13, 2022

View reviewed changes

lib/internal/modules/esm/fetch_module.js Outdated Show resolved Hide resolved

bmeck reviewed Feb 13, 2022

View reviewed changes

lib/internal/modules/esm/fetch_module.js Show resolved Hide resolved

JakobJingleheimer reviewed Feb 15, 2022

View reviewed changes

guybedford approved these changes Feb 16, 2022

View reviewed changes

benjamingr force-pushed the use-async-await-in-fetch-module branch 2 times, most recently from 67713f1 to ace0812 Compare February 16, 2022 18:05

benjamingr added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 16, 2022

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 16, 2022

This was referenced Feb 17, 2022

CI Reliability 2022-02-17 nodejs/reliability#205

Open

CI Reliability 2022-02-18 nodejs/reliability#206

Open

github-actions bot mentioned this pull request Feb 19, 2022

CI Reliability 2022-02-19 nodejs/reliability#207

Open

26 tasks

benjamingr added the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 19, 2022

benjamingr added request-ci Add this label to start a Jenkins CI on a PR. and removed commit-queue-failed An error occurred while landing this pull request using GitHub Actions. labels Feb 23, 2022

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 23, 2022

module: prefer async/await in https imports

b575c6d

benjamingr force-pushed the use-async-await-in-fetch-module branch from b3ef5cb to b575c6d Compare February 23, 2022 11:18

benjamingr added the request-ci Add this label to start a Jenkins CI on a PR. label Feb 23, 2022

github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Feb 23, 2022

benjamingr added the commit-queue Add this label to land a pull request using GitHub Actions. label Feb 23, 2022

nodejs-github-bot added commit-queue-failed An error occurred while landing this pull request using GitHub Actions. and removed commit-queue Add this label to land a pull request using GitHub Actions. labels Feb 23, 2022

benjamingr closed this Feb 23, 2022

benjamingr added a commit that referenced this pull request Feb 23, 2022

module: prefer async/await in https imports

7efef74

PR-URL: #41950 Fixes: #41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

sxa pushed a commit to sxa/node that referenced this pull request Mar 7, 2022

module: prefer async/await in https imports

96510b3

PR-URL: nodejs#41950 Fixes: nodejs#41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

sxa mentioned this pull request Mar 8, 2022

v17.7.0 release proposal #42254

Merged

xtx1130 pushed a commit to xtx1130/node that referenced this pull request Apr 25, 2022

module: prefer async/await in https imports

47d4289

PR-URL: nodejs#41950 Fixes: nodejs#41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

juanarbol added the backport-blocked-v16.x label May 30, 2022

juanarbol added backport-requested-v16.x and removed backport-blocked-v16.x labels May 30, 2022

targos removed the backport-requested-v16.x label Jul 19, 2022

targos pushed a commit that referenced this pull request Jul 19, 2022

module: prefer async/await in https imports

cb18e8e

PR-URL: #41950 Fixes: #41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

targos pushed a commit that referenced this pull request Jul 31, 2022

module: prefer async/await in https imports

abda05a

PR-URL: #41950 Fixes: #41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

targos mentioned this pull request Aug 3, 2022

v16.17.0 release proposal #44098

Merged

guangwong pushed a commit to noslate-project/node that referenced this pull request Oct 10, 2022

module: prefer async/await in https imports

b48c310

PR-URL: nodejs/node#41950 Fixes: nodejs/node#41950 Reviewed-By: Guy Bedford <[email protected]> Reviewed-By: James M Snell <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module: prefer async/await in https imports #41950

module: prefer async/await in https imports #41950

benjamingr commented Feb 12, 2022 •

edited

Loading

nodejs-github-bot commented Feb 12, 2022

GeoffreyBooth commented Feb 13, 2022

GeoffreyBooth left a comment

benjamingr commented Feb 13, 2022

bmeck commented Feb 13, 2022

benjamingr commented Feb 13, 2022

bmeck commented Feb 13, 2022

benjamingr commented Feb 13, 2022

JakobJingleheimer left a comment

JakobJingleheimer Feb 15, 2022

benjamingr commented Feb 16, 2022

guybedford left a comment

benjamingr commented Feb 16, 2022

nodejs-github-bot commented Feb 16, 2022

nodejs-github-bot commented Feb 18, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

benjamingr commented Feb 23, 2022

juanarbol commented May 30, 2022

	* It caches either the promise or the cahce entry since import.meta.url needs
	* It caches either the promise or the cache entry since import.meta.url needs

module: prefer async/await in https imports #41950

module: prefer async/await in https imports #41950

Conversation

benjamingr commented Feb 12, 2022 • edited Loading

nodejs-github-bot commented Feb 12, 2022

GeoffreyBooth commented Feb 13, 2022

GeoffreyBooth left a comment

Choose a reason for hiding this comment

benjamingr commented Feb 13, 2022

bmeck commented Feb 13, 2022

benjamingr commented Feb 13, 2022

bmeck commented Feb 13, 2022

benjamingr commented Feb 13, 2022

JakobJingleheimer left a comment

Choose a reason for hiding this comment

JakobJingleheimer Feb 15, 2022

Choose a reason for hiding this comment

benjamingr commented Feb 16, 2022

guybedford left a comment

Choose a reason for hiding this comment

benjamingr commented Feb 16, 2022

nodejs-github-bot commented Feb 16, 2022

nodejs-github-bot commented Feb 18, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

nodejs-github-bot commented Feb 23, 2022

benjamingr commented Feb 23, 2022

juanarbol commented May 30, 2022

benjamingr commented Feb 12, 2022 •

edited

Loading