Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

incompatible_applicable_licenses: Add global attribute "applicable_licenses" #10687

Closed
aiuto opened this issue Jan 30, 2020 · 29 comments
Closed
Assignees
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Configurability platforms, toolchains, cquery, select(), config transitions type: feature request

Comments

@aiuto
Copy link
Contributor

aiuto commented Jan 30, 2020

This flag enables a new attribute applicable_licenses to be globally available on any rule. It also enables a package() level attribute default_applicable_licenses.

Flag: --incompatible_applicable_licenses

Motivation: #7444 - we are pulling the existing licenses rule out of Bazel and creating a Starlark based rule set that adds more flexibility. Also see:
License Checking in Bazel

Migration
If your rules define an attribute applicable_licenses, you must rename that.
We expect to flip this flag in Bazel 4.x

@aiuto aiuto added P1 I'll work on this now. (Assignee required) team-Configurability platforms, toolchains, cquery, select(), config transitions incompatible-change Incompatible/breaking change labels Jan 30, 2020
@aiuto aiuto self-assigned this Jan 30, 2020
bazel-io pushed a commit that referenced this issue Jan 31, 2020
RELNOTES:
Create the incompatibleApplicableLicenses flag.
We plan to flip this from false to true in Bazel 4.x.
Implementation to follow.

#10687
#7444

PiperOrigin-RevId: 292603753
@aiuto aiuto added P2 We'll consider working on this in future. (Assignee optional) and removed P1 I'll work on this now. (Assignee required) labels Feb 20, 2020
@keith
Copy link
Member

keith commented Mar 8, 2020

Is this actively being worked on?

@aiuto
Copy link
Contributor Author

aiuto commented Mar 8, 2020 via email

@keith
Copy link
Member

keith commented Mar 8, 2020

Great thanks for the update!

@keith
Copy link
Member

keith commented Mar 30, 2020

@aiuto any updated estimate on when the macros repo will be open sourced?

@aiuto
Copy link
Contributor Author

aiuto commented Mar 31, 2020 via email

@ulfjack
Copy link
Contributor

ulfjack commented Apr 15, 2020

We needed to handle licenses in preparation for our first public release, and since this wasn't ready, I had to design my own license handling infrastructure. These are our requirements so far:

  • Generate a list of projects and license URLs.
  • Generate a directory or zip of all applicable licenses.
  • Also handle notice files from downstream projects.
  • Handle compile-time dependencies (both host and target configuration).

The documentation currently primarily documents what your proposed rules look like, but I'm more interested in what the intended interface between Bazel and the licensing rules looks like. From what I've seen so far, I'm not sure the Google rules will work for us, and I expect to continue to use our own.

@aiuto
Copy link
Contributor Author

aiuto commented Apr 15, 2020

Hi Ulf: FYI, I am days away from releasing the work so far to bazelbuild/rules_license. Just waiting for the last launch bit to flip. Once that happens, I can more quickly iterate on the examples there and canonical license kinds.

@aiuto
Copy link
Contributor Author

aiuto commented Apr 15, 2020 via email

@aiuto
Copy link
Contributor Author

aiuto commented Jun 10, 2020 via email

@keith
Copy link
Member

keith commented Jun 10, 2020

After just a brief look I don't completely understand how the pieces fit together, but I'm happy to look at some changes to try and get a better sense

@aiuto
Copy link
Contributor Author

aiuto commented Jun 11, 2020 via email

@jheaff1
Copy link
Contributor

jheaff1 commented Aug 16, 2021

Any update on this?

@adam-azarchs
Copy link
Contributor

It isn't clear from this issue, the design doc, or the rules_license repository what the current status is. I think it might be safe to start building stuff around the rules (e.g. our own list of license_kinds) but I'd rather not put a lot of effort into this if the public rules are way out of sync with the internal version so I'd have to overhaul everything for some future update.

Another thing that is unclear from the design document is whether this is ever expected to handle non-vendored third-party dependencies. In particular, what is going to be badly needed to make this useful for people who don't just vendor everything into their monorepo are updates to various repository fetching rules. npm_install would be a particularly important and complex example use case. If there is ever going to be built-in support from the official rules_nodejs for this, I would want to make sure that whatever I build for us internally stays at least mostly compatible with it. Other examples would include e.g. go_respository, pip_install, maven_install, and so on. Obviously can be done (with varying degrees of ease of use and reliability) but not clear whether there's plans to actually do it.

@aiuto
Copy link
Contributor Author

aiuto commented Oct 14, 2021 via email

@adam-azarchs
Copy link
Contributor

Like the rest of the Bazel project, we are focused on precise specification and repeatable mechanisms. Autoclassification can return results which are at some confidence level <100%, and worse, change each time the classification corpus changes. I'll let the experts in that domain work on that problem.

Sorry, I of course understand that license classification is outside the scope of rules_license per se. If a repository rule fetches a single package covered by a single license (usually the case for e.g. http_archive), splicing in a license specifier into the repo is not unreasonable and I wouldn't really expect an automated solution in those cases.

What I'm asking about is somewhat more limited in scope: for repository rules which are wrapping some other package ecosystem that already has a well-established and adhered-to mechanism for precisely specifying the license for a package (e.g. npm or maven), would there be plans to extend the corresponding repository rules (e.g. npm_install or maven_install) to surface that information, at least in the cases where there was no uncertainty about the license that was involved (that is the license name is an exact match for a SPDX identifier)? This would be particularly important for rules like npm_install where the fetched repository actually consists of potentially several thousand individual packages, each with their own license.

@ulfjack
Copy link
Contributor

ulfjack commented Oct 25, 2021

Questions:

  • Should an alias rule have a license?
  • Is there a way to disable inheriting the package-level default_applicable_licenses attribute for license rules? Right now, if you use default_applicable_licenses on a package that contains the license rule that it refers to, it creates a cyclical dependency because license rules also inherit the attribute.

@ulfjack
Copy link
Contributor

ulfjack commented Oct 25, 2021

Another problem I have is that the collection is unreliable. For example, the example rules only have a small list of propagating attributes, like srcs and deps, which is potentially missing a lot of dependencies, e.g., through resources. Also, if a rule pulls in an output file of another rule directly, the example aspect doesn't propagate to the generating rule (this can be fixed with apply_to_generating_rules = True).

@aiuto
Copy link
Contributor Author

aiuto commented Oct 29, 2021 via email

@aiuto
Copy link
Contributor Author

aiuto commented Oct 29, 2021 via email

@aiuto
Copy link
Contributor Author

aiuto commented Oct 29, 2021 via email

@ulfjack
Copy link
Contributor

ulfjack commented Oct 29, 2021

Also, there are different use cases requiring different attributes for licenses. In many cases, it's sufficient to know the type of license (like bsd-3 or apache-2), but we also have cases where we have to generate a 'software bill of materials'-style list with full license files, notices, and URLs. We are able to automatically (using heuristics) extract this information from npm sources.

@ulfjack
Copy link
Contributor

ulfjack commented Oct 29, 2021

The reason why we get the cyclical dependencies is that our license rule doesn't have a license_kinds attribute, as required by Bazel (and not documented anywhere?). See RuleClass.java:

  public boolean isBazelLicense() {
    return name.equals("_license") && hasAttr("license_kinds", BuildType.LABEL_LIST);
  }

@ulfjack
Copy link
Contributor

ulfjack commented Oct 29, 2021

It is apparently documented, so my fault for not seeing that earlier.

@ulfjack
Copy link
Contributor

ulfjack commented Oct 29, 2021

And I'm not able to get it to work. The https://github.com/bazelbuild/rules_license/blob/main/rules/license.bzl#L100 uses a macro to explicitly override applicable_licenses with an empty list, and that works, but I wasn't able to get it to work just by naming the license rule _license and adding the license_kinds attribute.

@adam-azarchs
Copy link
Contributor

Somewhat related to the propagation of the aspect across attributes is that there are different kinds of dependencies with different license propagation requirements. For example, cmake has a license, and you'd want to know about that license if you ended up packaging parts of cmake in your final artifacts, but if you were just using it for an invocation of rules_foreign_cc then you (probably) wouldn't need to propagate its license into the output of that invocation. The license for Intel MKL is another important weird case (I'm not a lawyer, so I'm not going to say anything about why). It would be helpful if at least some of that nuance could be captured in the rules, though of course be expected to cover every scenario like that (for example g++, where its license should propagate if you use -static-libstdc++ but not if you don't).

@fmeum
Copy link
Collaborator

fmeum commented Jan 15, 2022

@aiuto I tried to adopt rules_license in a ruleset, but encountered two blockers that I listed in bazelbuild/rules_license#9 that are fixed by bazelbuild/rules_license#7 and bazelbuild/rules_license#8. If these were fixed, rulesets could start to use rules_license to add license annotations.

@adam-azarchs
Copy link
Contributor

According to the design doc, it's expected that every organization will fork rules_license. However, generally-applicable changes like the above would be nice to get into the "official" version to keep things from diverging too far. Certainly the private visibility for the SPDX license set makes them useless (since there's nothing else in that package). For license rules in I think the vast majority of cases you'd want the rule to be visible to at least sub-packages - there's a decent argument to be made for the macro to set the visibility to :__subpackages__ by default.

Related to that, as I've been rolling these rules out I've found it would be convenient if default_applicable_licenses were available on workspace as well as package, since in a majority of cases (outside of Google, anyway), a bazel workspace corresponds to a git repository, which usually has a single license file in the root which applies to everything in the repo (possibly with a few exceptions here and there). I do however recognize that that might be tricky to get working because as far as I can tell the content of WORKSPACE files is more or less ignored for external repositories (though maybe bzlmod will be changing the situation there? It would be nice if bzlmod had some affordances for doing that, though last I heard that wasn't part of the plan).

@fmeum
Copy link
Collaborator

fmeum commented Jan 17, 2022

According to the design doc, it's expected that every organization will fork rules_license.

This point in the doc might warrant further clarification. As I understand it, many organizations will fork or vendor rules_license. However, rulesets and other open-source dependencies would have to be able to refer to a common set of license_kind definitions.

I do however recognize that that might be tricky to get working because as far as I can tell the content of WORKSPACE files is more or less ignored for external repositories (though maybe bzlmod will be changing the situation there? It would be nice if bzlmod had some affordances for doing that, though last I heard that wasn't part of the plan).

I think that bzlmod is flexible enough to support this use case as is: If rules_license defines appropriate module extensions, modules could use those to declare their default licenses and rules_license would be able to associate them with the modules as it will have visibility into the entire module dependency graph.

luca-digrazia pushed a commit to luca-digrazia/DatasetCommitsDiffSearch that referenced this issue Sep 4, 2022
    RELNOTES:
    Create the incompatibleApplicableLicenses flag.
    We plan to flip this from false to true in Bazel 4.x.
    Implementation to follow.

    bazelbuild/bazel#10687
    bazelbuild/bazel#7444

    PiperOrigin-RevId: 292603753
@aiuto aiuto removed incompatible-change Incompatible/breaking change wip labels Nov 10, 2022
@aiuto
Copy link
Contributor Author

aiuto commented Nov 10, 2022

I had forgotten this issue, since it was about the flag to add the feature. The feature was added and the flag removed long ago, so I am going to close this issue. Let's discuss more of what rules_license does in that repository rather than this thread.

To answer some of the points from above.

  • many of the issues/PR mentioned are now done
  • I am working on a unified mechanism to splice license and package information from upstream repos like Maven into the BUILD graph. I have a working prototype, but bzlmod breaks it, so more engineering is needed.
  • SBOM work is changing the whole face.

@aiuto aiuto closed this as completed Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P2 We'll consider working on this in future. (Assignee optional) team-Configurability platforms, toolchains, cquery, select(), config transitions type: feature request
Projects
None yet
Development

No branches or pull requests

7 participants