Does not work with latest version of yarn

[gigaSproule]

I've tried upgrading to the latest version of yarn from yarn v1, but it seems that the licenses command has been removed from yarn core. Obviously, no command means no data, but I'm not seeing any mention of yarn 2/3 not being supported. Is there something that I'm missing?

This is the stacktrace from the run within the licensefinder/license_finder:7.0.1 docker container:

/usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/package_managers/yarn.rb:20:in `current_packages': Command 'yarn licenses list --json' failed to execute:  (RuntimeError)
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/package_manager.rb:105:in `current_packages_with_relations'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `each'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `flat_map'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/scanner.rb:42:in `active_packages'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/core.rb:84:in `current_packages'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/core.rb:79:in `decision_applier'
	from /usr/share/rvm/rubies/ruby-3.1.1/lib/ruby/3.1.0/forwardable.rb:2[32](https://gitlab.com/bmi-digital/eslint-config/-/jobs/2510457490#L32):in `any_packages?'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:17:in `block in any_packages?'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `map'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/license_aggregator.rb:15:in `any_packages?'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/lib/license_finder/cli/main.rb:119:in `action_items'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor.rb:[39](https://gitlab.com/bmi-digital/eslint-config/-/jobs/2510457490#L39)2:in `dispatch'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
	from /usr/share/rvm/gems/ruby-3.1.1/gems/license_finder-7.0.1/bin/license_finder:6:in `<top (required)>'
	from /usr/share/rvm/gems/ruby-3.1.1/bin/license_finder:25:in `load'
	from /usr/share/rvm/gems/ruby-3.1.1/bin/license_finder:25:in `<main>'
	from /usr/share/rvm/gems/ruby-3.1.1/bin/ruby_executable_hooks:22:in `eval'
	from /usr/share/rvm/gems/ruby-3.1.1/bin/ruby_executable_hooks:22:in `<main>'

Yarn version is 3.2.1, node version is 14.19.1.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this. Unfortunately, the Pivotal Tracker project is private so you may be unable to view the contents of the story.

The labels on this github issue will be updated when the story is started.

[gigaSproule]

I've since found that license_finder is meant to add a plugin

LicenseFinder/lib/license_finder/package_managers/yarn.rb

Line 72 in 4d2390f

"#{yarn_plugin_production_command}yarn install && yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-licenses/#{yarn_licenses_plugin_version}/bundles/@yarnpkg/plugin-licenses.js"

, but it doesn't seem to do that.

When I added it myself, it then didn't error, but I get this output:

LicenseFinder::NPM: �is active
LicenseFinder::Yarn: �is active

No dependencies recognized!

I've tried it with pnp node linker and node_modules.

Having dug into the code a bit more, it seems like this will only work for node_modules, but it can't even get that far because the JSON produced by the plugin is different to the one produced by yarn v1.

The JSON structure from v1 returns a head and a body inside of data, where head and body are zipped to create a well structured JSON object of Name, Version, License, URL, VendorUrl, VendorName.
Formatted example:

{
  "type": "table",
  "data": {
    "head": ["Name", "Version", "License", "URL", "VendorUrl", "VendorName"],
    "body": [
      [
        "eslint-plugin-security",
        "1.5.0",
        "Apache-2.0",
        "git+https://github.com/nodesecurity/eslint-plugin-security.git",
        "https://github.com/nodesecurity/eslint-plugin-security#readme",
        "Node Security Project"
      ]
    ]
  }
}

With the plugin that tries to get installed, the JSON structure is actually multiple lines of JSON, 1 for each license group, with the structure value (which is the license name), children. children is then a list of the dependencies, with the key being the dependency name + version and the value being a nested object containing url, vendorName and vendorUrl.
Formatted example:

{
  "value": "Apache-2.0",
  "children": {
    "eslint-plugin-security@npm:1.5.0": {
      "value": {
        "locator": "eslint-plugin-security@npm:1.5.0",
        "descriptor": "eslint-plugin-security@npm:^1.5.0"
      },
      "children": {
        "url": "git+https://github.com/nodesecurity/eslint-plugin-security.git",
        "vendorName": "Node Security Project",
        "vendorUrl": "https://github.com/nodesecurity/eslint-plugin-security#readme"
      }
    }
  }
}

With the v1 structure, it's going through the if block (

LicenseFinder/lib/license_finder/package_managers/yarn.rb

Line 28 in 4d2390f

if json_objects.last['type'] == 'table'

) which parses the JSON properly, but for the plugin structure, it can only try the regex approach (

LicenseFinder/lib/license_finder/package_managers/yarn.rb

Line 33 in 4d2390f

json_objects.each do |json_object|

), but as it doesn't contain a data property, it will never match.

[xtreme-shane-lattanzio]

hey @gigaSproule ! I think we put a PR in for this months ago here: #891. It should be making adjustments based on your yarn version. If you look in the yarn.rb file, there is logic to check the version based on yarn -v and should run:

"#{yarn_plugin_production_command}yarn install && yarn plugin import https://raw.githubusercontent.com/mhassan1/yarn-plugin-licenses/#{yarn_licenses_plugin_version}/bundles/@yarnpkg/plugin-licenses.js"

if the version is >1.

The production command only gets adding to the yarn install if ignored groups includes devDependencies by the look of things:

    def yarn_plugin_production_command
      return '' if @ignored_groups.nil?

      @ignored_groups.include?('devDependencies') ? 'yarn plugin import workspace-tools && yarn workspaces focus --all --production && ' : ''
    end

Does this help at all? Let me know!

[gigaSproule]

@xtreme-shane-lattanzio sorry for taking so long to get back to you. I've tried upgrading license_finder and trying again, but it seems there are no new versions, so I can only assume that the PR you are mentioning was part of the 7.0.1 release, which I originally tried this with.

[xtreme-shane-lattanzio]

Hey @gigaSproule ! I think you are right. When we tested this I think it was more about fixing yarn2 causing yarn1 issues but we never tested with a yarn 2 project. I am confirming this but we may need a PR to not only change the yarn command on the version but also change the json parsing

[xtreme-shane-lattanzio]

@gigaSproule Alright I tried on this dummy repo and something is definitely wrong: https://github.com/yarnpkg/example-yarn-package

I ran with v1 and got the full output. When updating to 3.2.2, I get

No dependencies recognized!

This looks to be more than a parsing issue because this output is the stdout from the yarn licenses list --json command which seems much smaller than v1 (or maybe I need a new format for the lock file?)

{"value":"BSD-2-Clause","children":{"example-yarn-package@workspace:.":{"value":{"locator":"example-yarn-package@workspace:.","descriptor":"example-yarn-package@workspace:."},"children":{"url":"github.com/yarnpkg/example-yarn-package"}}}}
{"value":"BSD-3-Clause","children":{"jest-cli@npm:15.1.1":{"value":{"locator":"jest-cli@npm:15.1.1","descriptor":"jest-cli@npm:15.1.1"},"children":{"url":"https://github.com/facebook/jest","vendorUrl":"http://facebook.github.io/jest/"}}}}
{"value":"MIT","children":{"lodash@npm:4.16.2":{"value":{"locator":"lodash@npm:4.16.2","descriptor":"lodash@npm:^4.16.2"},"children":{"url":"https://lodash.com/","vendorUrl":"https://lodash.com/"}}}}

I'm not sure when I will have time to look into this but if you can put up a PR, I can take a look and it may be faster. Thanks for bringing this up either way.

[gigaSproule]

Yeah, I saw the same sort of thing. I should have pasted the output from our project, so sorry about that. I'll try it again and update here to ensure I'm getting the same format.

As with you, I'm time poor 😉 I'll have to see if I can get a bit of time together to have a crack at it.

[xtreme-shane-lattanzio]

I took a stab at it with some guesses. Let me know what you think! #936

[Kurt-von-Laven]

I went into more depth about this on the PR, but basically transitive dependencies should be checked by passing --recursive to the Yarn plugin.

[Kurt-von-Laven]

@xtreme-shane-lattanzio, the significant functionality you added seems worthy of a release if time allows.

[kmorinaka]

This still doesn't work for yarn 4. Can the yarn_licenses_plugin_version be updated to v0.13.1 - the latest version of yarn-plugin-licenses to support yarn 4?

[Kurt-von-Laven]

I suggest filing a new issue for greater visibility and installing the needed plugin yourself in the meantime.