yarn workspaces #57

qknight · 2018-06-09T22:43:10Z

i've discovered yarn workspaces in the project https://gitlab.coko.foundation/editoria/editoria/tree/master and i assume this is currently not handled by yarn2nix.

the problem is this:

editoria/package.json (with editoria/yarn.lock) can be easily transformed into a node_package created by yarn2nix
editoria/packages/editoria-app/package.json is a subdirectory/subproject which also uses the yarn.lock from the top-level but then generates a different node_modules output as it is based on a different package.json

i'm not sure how to cover this use-case with yarn2nix but hopefully only requires a little patch for yarn2nix. ideas?

The text was updated successfully, but these errors were encountered:

qknight · 2018-06-09T23:19:34Z

interestingly:

Note that Workspaces don’t have their own yarn.lock files, and the root yarn.lock contains all the dependencies for all the Workspaces. When you want to change a dependency inside a Workspace, the root yarn.lock will be changed as well as the Workspace’s package.json.

so i tried to use the global yarn.lock and the yarn.nix, which both contain the webpack dependency to build a node_modules using yarn2nix. however, the produced node_modules from nix-build lacked the .bin/webpack as well as other webpack related directories which is already a difference compared to the outcome of yarn run statefully. bug?

still, the point is when using yarn (and not yarn2nix):

editoria/package.json + editoria/yarn.lock                       -> editoria/node_modules
editoria/packages/editoria-app/package.json + editoria/yarn.lock -> editoria/packages/editoria-app/node_modules

both of these two node_modules contain different stuff. this means we can't just convert the yarn.lock into a node_modules without taking packages.json into account. i don't know the implementation of yarn2nix yet, so maybe this is already taken into account but for now i assume that packages.json is only used to detect that the yarn.lock is outdated and needs a new yarn run.

update: above assumption is probably wrong. yarn2nix probably uses the package.json to also select the dependencies required.

qknight · 2018-06-10T10:32:52Z

top level package.json

the global editoria/package.json contains a workspace definition:

"workspaces": [
   "packages/*"
 ]

nested package.json

editoria/packages/editoria-app/package.json does not have the workspaces definition but still uses the nested projects from the top-level workspace:

editoria/packages/editoria-app $ nix-build
warning: Nix search path entry '$HOME/.nix-defexpr/channels' does not exist, ignoring
these derivations will be built:
  /nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv
  /nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv
building '/nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv'...
configuring
building
yarn config v1.6.0
success Set "yarn-offline-mirror" to "/nix/store/0xksa6xjnzsnp5s3wj7xkz002if8wpb3-offline".
Done in 0.07s.
yarn install v1.6.0
[1/4] Resolving packages...
error Couldn't find any versions for "pubsweet-editoria-authsome" that matches "^0.0.1" in our cache (possible versions are ""). This is usually caused by a missing entry in the lockfile, running Yarn without the --offline flag may help fix this issue.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
builder for '/nix/store/9hari9gxg6kdqfvj7q86z144z3difngb-Editoria-modules-1.2.0.drv' failed with exit code 1
cannot build derivation '/nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv': 1 dependencies couldn't be built
error: build of '/nix/store/apb7v10zazdjjbhr0i25nm62y6gw0xx1-pubsweet.drv' failed

looking into https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/, this is how dependency resolving works:

Packages like diff, pretty-format and the symlink to jest-matcher-utils were hoisted to the root
node_modules directory, making the installation faster and smaller. The package chalk however could
not be moved to the root because the root already depends on a different, incompatible version of chalk.

...

If you run code inside the jest-diff Workspace, it will be able to resolve all its dependencies:

require(‘chalk’) resolves to ./node_modules/chalk
require(‘diff’) resolves to ../../node_modules/diff
require(‘pretty-format’) resolves to ../../node_modules/pretty-format
require(‘jest-matcher-utils’) resolves to ../../node_modules/jest-matcher-utils that is a symlink to ../packages/jest-matcher-utils

which implies that we have to take the other node_modules folder into account also. but how?

workspace dependencies

turns out this was not enough as the workspace contains lots of nested libraries which have node_modules created for each also. so this node_module hierarchy basically is applied all over the place.

for now, this seems to be a far more complex issue then i expected it to be at first.

qknight · 2018-06-10T12:54:07Z

maybe this is the solution:

https://yarnpkg.com/lang/en/docs/cli/install/#toc-yarn-install-focus

Shallowly installs a packages’s sibling workspace dependencies underneath its node_modules folder. > This allows you to run that workspace without building the other workspaces it depends on.
Must be run inside an individual workspace in a workspaces project. Can not be run in a non-
workspaces project or at the root of a workspaces project.

update: it is not, see yarnpkg/yarn#5864 (comment)

qknight · 2018-06-12T11:36:01Z

i've been thinking to copy the node_modules path together as a post-yarn step using some code from https://github.com/ripeworks/workspace-pack

experiment manual copying

what i tried:

yarn --ignore-engines and then i have a global node_moduleseach of the subdirectories in packages/ also has an individual node_modules
- bookbuilder
- common
- dashboard
- editoria-app
- editoria-authsome
- jest-environment-db
- theme
- vivliostyle

next i removed all the other subprojects:

  rm -Rf bookbuilder  common  dashboard  editoria-authsome  jest-environment-db  theme  vivliostyle

i tried to start the server: cd editoria/packages/editoria-app/ and yarn server and i get:

yarn server
yarn run v1.7.0
$ pubsweet start
/tmp/editoria/node_modules/config/lib/config.js:948
    throw new Error("Cannot parse config file: '" + fullFilename + "': " + e3);
    ^

Error: Cannot parse config file: '/tmp/editoria/packages/editoria-app/config/default.js': Error: Cannot find module 'pubsweet-editoria-authsome'
    at Config.util.parseFile (/tmp/editoria/node_modules/config/lib/config.js:948:11)
    at /tmp/editoria/node_modules/config/lib/config.js:717:28
    at Array.forEach (<anonymous>)
    at /tmp/editoria/node_modules/config/lib/config.js:713:14
    at Array.forEach (<anonymous>)
    at Config.util.loadFileConfigs (/tmp/editoria/node_modules/config/lib/config.js:712:13)
    at new Config (/tmp/editoria/node_modules/config/lib/config.js:135:27)
    at Object.<anonymous> (/tmp/editoria/node_modules/config/lib/config.js:1782:31)
    at Module._compile (module.js:643:30)
    at Object.Module._extensions..js (module.js:654:10)
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

summary

assembling a global node_packages for editoria/packages/editoria-app means we have to not only take the local node_modules and the top-level node_modules into account but also all the other subprojects which makes it rather complicated.

qknight · 2018-06-13T12:54:04Z

using a 'hack' i was able to generate a standalone node_modules as described yarnpkg/yarn#5965 (comment) statefully (using yarn on the shell manually) but i wasn't able to adapt this hack into yarn2nix so far.

update: was able to 'update' a yarn.lock by removing all the bundled libraries:

[email protected]:
  version "1.0.10"
  resolved "https://registry.yarnpkg.com/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.10.tgz#1c595000f04a8897dfb85000892a0f4c33af86c3"
  dependencies:
    safe-buffer "^5.0.1"

"editoria-common@file:packages/common":
  version "0.0.3"
  dependencies:
    react "^16.2.0"
    react-bootstrap "0.32.1"
    react-router-dom "4.2.2"

now i get:

yarn install v1.6.0
[1/4] Resolving packages...
error Your lockfile needs to be updated, but yarn was run with `--frozen-lockfile`.
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
builder for '/nix/store/6pm1fgaxg7yi748aahgkww2bqrhz8mrq-editoria-modules-1.2.0.drv' failed with exit code 1
cannot build derivation '/nix/store/sdyp8z5rng97khl4cj4gy73j8pd907fs-pubsweet.drv': 1 dependencies couldn't be built
error: build of '/nix/store/sdyp8z5rng97khl4cj4gy73j8pd907fs-pubsweet.drv' failed

nightkr · 2018-06-13T23:08:42Z

I've hacked in support for this as of https://github.com/teozkr/yarn2nix/tree/de4ddd1d117af93e8874d4dd4d27d97f70d83328. The basic idea is that we build each
item in the workspace separately, and then yarn add the local path onto the dependee (overriding the intra-workspace dependency).

It's very hacky, breaks --frozen-lockfile (so you're not 100% guaranteed that the lockfile is up-to-date), and requires you to manually tell Nix about the dependency tree. On the other hand, it seems to correctly build Photo Garden's gateway component, and I'm working on porting the rest (see the .nix files in https://github.com/teozkr/yarn2nix/tree/de4ddd1d117af93e8874d4dd4d27d97f70d83328).

See nix-community#57 for the initial ticket. nix-community#58 adds the basic plumbing, but requires you to manually define the workspace. This will automatically import the "master" `package.json` and find all subpackages.

qknight · 2018-08-28T13:26:00Z

@teozkr been looking into what you did with #58 and #59 and it seems interesting for our work. yet i don't understand how to use your code. the test is a little help but i don't understand how to adapt it to my problem either.

a few questions:

how to select the primary 'workspace package' for which i want to build the node_modules folder (in my case editoria-app)?
note: the main application is also a package, see packages/editoria-app/ where the package.json states:
```
 "workspaces": [
    "packages/*"
  ]
```
what does libexec/editoria-monorepo/deps/ mean? what are deps and how to use them? i assumed that i only needed node_modules and not deps
why use the directory name libexec and not, say share?

nightkr · 2018-08-28T16:54:52Z

@qknight: To build a package from the workspace you'll want some variant of:

(mkYarnWorkspace {
  src = ./.;
}).editoria-app

deps contains the package.jsons of all dependencies, so that we can cache the result of yarn install even though the package contents change.

I have no idea what the libexec stuff is supposed to do, that seems to have been added after my changes.

qknight · 2018-08-28T21:22:24Z

@teozkr awesome, got it built. thanks so much! need to verify that it is working still. but the correct command was:

(mkYarnWorkspace {
  src = ./.;
}).Editoria

qknight mentioned this issue Jun 10, 2018

yarn workspaces Profpatsch/yarn2nix#16

Open

qknight mentioned this issue Jun 10, 2018

'yarn workspaces' break yarn2nix workflow yarnpkg/yarn#5965

Open

nightkr mentioned this issue Jun 15, 2018

Support for Yarn workspaces #58

Merged

zimbatm closed this as completed in #58 Jun 17, 2018

nightkr mentioned this issue Jun 26, 2018

Automatic Yarn Workspace import #59

Merged

tupakkatapa mentioned this issue Apr 11, 2023

wip: add lodestar support nix-community/ethereum.nix#210

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

yarn workspaces #57

yarn workspaces #57

qknight commented Jun 9, 2018 •

edited

Loading

qknight commented Jun 9, 2018 •

edited

Loading

qknight commented Jun 10, 2018 •

edited

Loading

qknight commented Jun 10, 2018 •

edited

Loading

qknight commented Jun 12, 2018

qknight commented Jun 13, 2018 •

edited

Loading

nightkr commented Jun 13, 2018

qknight commented Aug 28, 2018 •

edited

Loading

nightkr commented Aug 28, 2018 •

edited

Loading

qknight commented Aug 28, 2018 •

edited

Loading

yarn workspaces #57

yarn workspaces #57

Comments

qknight commented Jun 9, 2018 • edited Loading

qknight commented Jun 9, 2018 • edited Loading

qknight commented Jun 10, 2018 • edited Loading

top level package.json

nested package.json

workspace dependencies

qknight commented Jun 10, 2018 • edited Loading

qknight commented Jun 12, 2018

experiment manual copying

summary

qknight commented Jun 13, 2018 • edited Loading

nightkr commented Jun 13, 2018

qknight commented Aug 28, 2018 • edited Loading

nightkr commented Aug 28, 2018 • edited Loading

qknight commented Aug 28, 2018 • edited Loading

qknight commented Jun 9, 2018 •

edited

Loading

qknight commented Jun 9, 2018 •

edited

Loading

qknight commented Jun 10, 2018 •

edited

Loading

qknight commented Jun 10, 2018 •

edited

Loading

qknight commented Jun 13, 2018 •

edited

Loading

qknight commented Aug 28, 2018 •

edited

Loading

nightkr commented Aug 28, 2018 •

edited

Loading

qknight commented Aug 28, 2018 •

edited

Loading