module: expose `getPackageJSON` utility #55229

JakobJingleheimer · 2024-10-01T22:44:43Z

Finally got around to exposing one of the utilities we've long-discussed providing to users.

Currently, users (particularly library authors like yarn, who I believe originally requested this) have to re-implement this functionality.

I ran into this issue when building a codemod that needs to consume pjson.types / pjson.exports[…].types

JakobJingleheimer/correct-ts-specifiers#6

References:

nodejs-github-bot · 2024-10-01T22:44:49Z

Review requested:

@nodejs/loaders

doc/api/module.md

typings/internalBinding/modules.d.ts

codecov · 2024-10-02T00:17:12Z

Codecov Report

Attention: Patch coverage is 97.36842% with 4 lines in your changes missing coverage. Please review.

Project coverage is 88.04%. Comparing base (103b843) to head (864b98a).
Report is 190 commits behind head on main.

Files with missing lines	Patch %	Lines
src/node_modules.cc	84.21%	0 Missing and 3 partials ⚠️
lib/internal/modules/cjs/loader.js	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #55229      +/-   ##
==========================================
- Coverage   88.23%   88.04%   -0.19%     
==========================================
  Files         651      652       +1     
  Lines      183863   186830    +2967     
  Branches    35824    35999     +175     
==========================================
+ Hits       162235   164501    +2266     
- Misses      14932    15562     +630     
- Partials     6696     6767      +71

Files with missing lines	Coverage Δ
lib/internal/modules/esm/resolve.js	`96.35% <100.00%> (-0.19%)`	⬇️
lib/internal/modules/package_json_reader.js	`100.00% <100.00%> (ø)`
lib/module.js	`100.00% <100.00%> (ø)`
src/node_modules.h	`100.00% <ø> (ø)`
lib/internal/modules/cjs/loader.js	`96.26% <90.90%> (-1.16%)`	⬇️
src/node_modules.cc	`78.65% <84.21%> (+0.55%)`	⬆️

... and 144 files with indirect coverage changes

richardlau · 2024-10-06T13:42:26Z

a require('../') in node-gyp where there is no index.js in the parent dir, so I'm not sure how this was ever working? does require() read from package.json (ex its "main" field)?

Yes, if neither ../index.js, ../index.json nor ../index.node was found. See https://nodejs.org/docs/latest/api/modules.html#all-together.

fixup: de-lint

This reverts commit 395cfa6.

RedYetiDev · 2024-10-09T00:17:51Z

doc/api/module.md

+
+* `startLocation` {string} A fully resolved FS path or file URL to start looking
+* `everything` {boolean} Whether to return the full contents of the found package.json
+* Returns: {Object | undefined}


Suggested change

* Returns: {Object | undefined}

* Returns: {Object|undefined}

Ditto to the rest

It's the way the docs are formatted. There is no space between the |.

Does it actually matter or is it a style/consistency issue?

I don't think the linter will throw an error, but it's documented in the Style Guide (https://github.com/nodejs/node/blob/main/doc/README.md#function-arguments-and-returns)

Yeah let's keep them consistent, i.e. without spaces

RedYetiDev · 2024-10-09T00:18:15Z

doc/api/module.md

+  * data: {Object}
+    * name: {string | undefined}
+    * type: {string | undefined}
+    * exports: string | string\[] | Record\<string, unknown> | undefined


Is this a valid type?

I dunno if the validator considers it one, but it absolutely valid 🙂

Note that we are not using TS nor JSDoc syntax, so it doesn't really make sense to claim it's "valid syntax" if the validator disagrees

Suggested change

* exports: string | string\[] | Record\<string, unknown> | undefined

* `exports`: {string|string\[]|Object|undefined}

lib/internal/modules/package_json_reader.js

RedYetiDev · 2024-10-09T00:22:41Z

lib/internal/modules/package_json_reader.js

+  if (everything) {
+    const result = modulesBinding.getNearestRawParentPackageJSON(startPath);
+
+    return {


Does this also need to be null prototyped

@aduh95 please advise 🙏

no, it's user-facing

aduh95 · 2024-10-10T08:56:25Z

doc/api/module.md

+  * data: {Object}
+    * name: {string | undefined}
+    * type: {string | undefined}


ditto with the rest

Suggested change

* data: {Object}

* name: {string | undefined}

* type: {string | undefined}

* `data`: {Object}

* `name`: {string|undefined}

* `type`: {string|undefined}

aduh95 · 2024-10-10T08:56:58Z

doc/api/module.md

+    * …
+  * path: {string}
+
+Retreives the contents and location of the package.json closest to the supplied `startLocation`;


Suggested change

Retreives the contents and location of the package.json closest to the supplied `startLocation`;

Retreives the contents and location of the `package.json` closest to the supplied `startLocation`;

aduh95 · 2024-10-10T08:57:39Z

doc/api/module.md

+  * path: {string}
+
+Retreives the contents and location of the package.json closest to the supplied `startLocation`;
+this behaves identically to node's own lookup and consumption of package.json for a given module.


Suggested change

this behaves identically to node's own lookup and consumption of package.json for a given module.

this behaves identically to Node.js own lookup and consumption of `package.json` for a given module.

aduh95 · 2024-10-10T09:01:43Z

lib/internal/modules/package_json_reader.js

+      ...(name != null && { __proto__: null, name }),
+      ...(main != null && { __proto__: null, main }),
+      ...(type != null && { __proto__: null, type }),


null prototype is not useful here, spread only copies own properties

Suggested change

...(name != null && { __proto__: null, name }),

...(main != null && { __proto__: null, main }),

...(type != null && { __proto__: null, type }),

...(name != null && { name }),

...(main != null && { main }),

...(type != null && { type }),

arcanis · 2024-10-10T09:10:45Z

Currently, users (particularly library authors like yarn, who I believe originally requested this) have to re-implement this functionality.

Locating the closest package.json isn't a concern for us since the PnP map is already aware of the location of all packages on disk. We're interested by the various resolution utilities private to the Node implementation. Basically things from here¹:

https://github.com/yarnpkg/berry/tree/master/packages/yarnpkg-pnp/sources/node

As for this function specifically, a C++ implementation means it doesn't go through fs, so it doesn't go through any virtual filesystem layer that have been setup, so we'll have to patch it as well. I'd really prefer if Node.js could have one and only one interface through which filesystem calls are performed 😕

and while some of them are loading files from the FS, we care more about what they do with it - it'd be perfectly ok if the exposed utilities required the user to provide the package.json content, for example ↩

JakobJingleheimer · 2024-10-10T09:51:01Z

As for this function specifically, a C++ implementation means it doesn't go through fs

This is already the case for node's package.json reading etc

arcanis · 2024-10-10T09:56:46Z

This is already the case for node's package.json reading etc

Which is private logic. And that's exactly why we have the loader hooks: to rewrite parts of the resolution in a composable way (so that hook X can apply its changes on top of those performed by hook Y). I guess we don't want utility hooks, so I think the design of said utilities should avoid relying on "hidden inputs" (the filesystem).

JakobJingleheimer · 2024-10-10T10:01:01Z

@arcanis I'm not sure what you're getting at. It sounds like you're suggesting a drastic overhaul of node's pjson find behaviour. I'm not necessarily opposed to that, but that's super out of scope for this, and those sorts of under-the-hood changes can happen transparently for this utility.

I was totally fine merely exposing a "find" utility. That was my original plan (and then to perhaps subsequently expose a get which consumed the find), but was strongly dissuade by @anonrig due to the C++ implications. If those are not valid (or we just don't care about them), I can switch back to just the find.

That would be extremely frustrating given how much time this current one took (especially due to the "can you also tidy up …" requests, which actually took the vast majority of time for this PR).

arcanis · 2024-10-10T10:27:19Z

To be clear, my primary point is: how does this work with hooks which don't work on the filesystem? Let's say I have --loader zip --loader ts and ts happens to call getPackageJSON, what will happen? It's a valid and preexisting use case, and it should be taken into consideration in the design.

It sounds like you're suggesting a drastic overhaul of node's pjson find behaviour.

I don't suggest a "drastic" overhaul: I don't mind that the native core resolution leverages native utilities. If you prefer that for performances, that's perfectly fine by me. That's why we have the hooks: to abstract all this logic in a pipeline that userland JS code can transparently integrate with (through nextResolve / nextFetch / etc).

The problem is when such native utilities are exposed through other access points. Then it becomes impossible to make them work with custom logic that hooks are supposed to allow, and they'll poison any other hook using them by making them incompatible as well.

What I'd tend to suggest would be to:

Keep the native implementation of the package crawl for the default resolver
But implement the same logic in JS just for the purpose of those published utilities, and use fs

It will lead to code duplication, but it'll ensure that the utilities will always have the exact same behavior as the default resolver on the Node.js version they run on (which wouldn't be the case if they were an independant package, for instance).

Alternatively, we need a reassurance that it'll always be possible to monkey-patch such utilities. But we all hate monkey-patching, and the hooks exist in part to make them a thing of the past.

JakobJingleheimer · 2024-10-10T10:38:08Z

I think all of those points, whilst valid, are not strictly relevant to this PR: those are under-the-hood details that can happen before or after, transparently.

This PR (or a "find" util) is only to expose the result of what node is doing. What and how node is doing it can change, and this would just continue to surface that.

JakobJingleheimer added 10 commits October 1, 2024 22:58

module: expose getPackageJSON utility

48ab696

fixup: convert path_value to string before fn casting to path

dcddd1e

fixup: remove obsolete reference to findNearestPackageJSON

98c72f4

fixup: add some tests cases

d4ede54

feat: adjust getNearestParentPackageJSON().path to pjsonPath

92f2c62

feat: conditionally include non-required fields only when they exist

b83707b

fixup: remove unnecessary JSON.parse() (it's already parsed)

e0beefa

test: add more cases

6e715ec

fix: provide missing pjsonPath for GetNearestRawParentPackageJSON

458dcca

fix: correct internal bindings type decs

903d201

JakobJingleheimer requested a review from anonrig October 1, 2024 22:44

nodejs-github-bot added lib / src Issues and PRs related to general changes in the lib or src directory. needs-ci PRs that need a full CI run. labels Oct 1, 2024

JakobJingleheimer added c++ Issues and PRs that require attention from people who are familiar with C++. semver-minor PRs that contain new features and should be released in the next minor version. labels Oct 1, 2024

fixup: remove unnecessary string manipulation

b567e9a

JakobJingleheimer commented Oct 1, 2024

View reviewed changes

doc/api/module.md Outdated Show resolved Hide resolved

JakobJingleheimer added 2 commits October 2, 2024 01:02

fixup: de-lint

ed4db64

this is awful. why does anyone want this lint rule

9a4b1d1

RedYetiDev reviewed Oct 1, 2024

View reviewed changes

typings/internalBinding/modules.d.ts Outdated Show resolved Hide resolved

RedYetiDev added module Issues and PRs related to the module subsystem. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. and removed lib / src Issues and PRs related to general changes in the lib or src directory. labels Oct 1, 2024

JakobJingleheimer added 2 commits October 2, 2024 09:25

fixup: correct yaml version needle

c23c8f1

fixup: americanize name

c865d55

This comment was marked as resolved.

Sign in to view

fixup: mangle md

a9bf393

JakobJingleheimer added 2 commits October 6, 2024 10:38

fixup: remove unnecessary cost for default value object instantiation

da29815

fixup: throw on relative statLocation with remediation instructions

b531497

JakobJingleheimer force-pushed the feat/module-utils-getFullPJSON branch from d76e8be to b531497 Compare October 6, 2024 09:02

JakobJingleheimer added 4 commits October 7, 2024 23:57

fixup: add test-case for require() resolving via package.json

eef2b60

fixup: update consumption of readPackage & read()

395cfa6

fixup: de-lint

Revert "fixup: update consumption of readPackage & read()"

841848f

This reverts commit 395cfa6.

fixus: restore return signature of jpsonReader.read

945c207

JakobJingleheimer force-pushed the feat/module-utils-getFullPJSON branch from cdc908a to 945c207 Compare October 8, 2024 20:39

RedYetiDev reviewed Oct 9, 2024

View reviewed changes

JakobJingleheimer added 4 commits October 9, 2024 12:53

fixup: null proto for read return

4adac4c

fixup: !== null → != null

0064379

fixup: spread with __proto__: null

eed739c

fixup: add exists to DeserializedPackageConfig

864b98a

JakobJingleheimer force-pushed the feat/module-utils-getFullPJSON branch from d65949a to 864b98a Compare October 9, 2024 21:43

JakobJingleheimer requested review from anonrig, aduh95 and marco-ippolito October 10, 2024 00:15

aduh95 reviewed Oct 10, 2024

View reviewed changes

JakobJingleheimer mentioned this pull request Oct 16, 2024

module: add findPackageJSON util #55412

Merged

nodejs-github-bot closed this in #55412 Oct 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

module: expose `getPackageJSON` utility #55229

module: expose `getPackageJSON` utility #55229

JakobJingleheimer commented Oct 1, 2024 •

edited

Loading

nodejs-github-bot commented Oct 1, 2024

codecov bot commented Oct 2, 2024 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

richardlau commented Oct 6, 2024 •

edited

Loading

RedYetiDev Oct 9, 2024

JakobJingleheimer Oct 9, 2024

RedYetiDev Oct 9, 2024

JakobJingleheimer Oct 9, 2024

RedYetiDev Oct 9, 2024

aduh95 Oct 10, 2024

RedYetiDev Oct 9, 2024

JakobJingleheimer Oct 9, 2024

aduh95 Oct 10, 2024 •

edited

Loading

RedYetiDev Oct 9, 2024

JakobJingleheimer Oct 9, 2024

aduh95 Oct 10, 2024

aduh95 Oct 10, 2024

aduh95 Oct 10, 2024

aduh95 Oct 10, 2024

aduh95 Oct 10, 2024

arcanis commented Oct 10, 2024 •

edited

Loading

JakobJingleheimer commented Oct 10, 2024

arcanis commented Oct 10, 2024 •

edited

Loading

JakobJingleheimer commented Oct 10, 2024 •

edited

Loading

arcanis commented Oct 10, 2024

JakobJingleheimer commented Oct 10, 2024

	* Returns: {Object \| undefined}
	* Returns: {Object\|undefined}

	* exports: string \| string\[] \| Record\<string, unknown> \| undefined
	* `exports`: {string\|string\[]\|Object\|undefined}

	Retreives the contents and location of the package.json closest to the supplied `startLocation`;
	Retreives the contents and location of the `package.json` closest to the supplied `startLocation`;

	this behaves identically to node's own lookup and consumption of package.json for a given module.
	this behaves identically to Node.js own lookup and consumption of `package.json` for a given module.

module: expose getPackageJSON utility #55229

module: expose getPackageJSON utility #55229

Conversation

JakobJingleheimer commented Oct 1, 2024 • edited Loading

nodejs-github-bot commented Oct 1, 2024

codecov bot commented Oct 2, 2024 • edited Loading

Codecov Report

This comment was marked as resolved.

This comment was marked as resolved.

richardlau commented Oct 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aduh95 Oct 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

arcanis commented Oct 10, 2024 • edited Loading

Footnotes

JakobJingleheimer commented Oct 10, 2024

arcanis commented Oct 10, 2024 • edited Loading

JakobJingleheimer commented Oct 10, 2024 • edited Loading

arcanis commented Oct 10, 2024

JakobJingleheimer commented Oct 10, 2024

module: expose `getPackageJSON` utility #55229

module: expose `getPackageJSON` utility #55229

JakobJingleheimer commented Oct 1, 2024 •

edited

Loading

codecov bot commented Oct 2, 2024 •

edited

Loading

richardlau commented Oct 6, 2024 •

edited

Loading

aduh95 Oct 10, 2024 •

edited

Loading

arcanis commented Oct 10, 2024 •

edited

Loading

arcanis commented Oct 10, 2024 •

edited

Loading

JakobJingleheimer commented Oct 10, 2024 •

edited

Loading