feat(license): improve license normalization #7131

pbaumard · 2024-07-09T15:04:08Z

Description

Normalize "+", "-only" and "-or-later" suffixes
Many new mappings from oss-review-toolkit
Space normalization, including newlines
Normalize US / UK spelling for license / licence.
Remove "THE " prefix
Remove common suffixes: " LICENSE", " LICENSED", -"LICENSE" and "-LICENSED"
Add MIT-0
Add tests

Related issues

Close feat(license): Improve license normalization #7189

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

- Remove "THE " prefix - Remove "LICENSE " suffix - Fix lowercase key - Add BSD LICENSE 3 - Add ECLIPSE PUBLIC LICENSE 1.0 - Add MIT-0 - Add tests

CLAassistant · 2024-07-09T15:04:14Z

All committers have signed the CLA.

CLAassistant · 2024-07-09T15:04:14Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

DmitriyLewen

Hello @pbaumard
Thanks for your work!

LGTM.
Can you fix tests?

pkg/licensing/normalize_test.go

pkg/licensing/normalize.go

DmitriyLewen

Hello @pbaumard
Sorry for delay.

Left comments. Take a look when you have time.

pkg/licensing/normalize.go

DmitriyLewen · 2024-07-24T09:48:01Z

pkg/licensing/normalize.go

+var mapping = make(map[string]expression.SimpleExpr)
+
+func addMap(name, key string, hasPlus bool) {
+	license := normalizeKeyAndSuffix(name)


IIUC licenses are already normalized.
Do we need this?

The license from normalizeKeyAndSuffix is only used for the following assertion with the panic error.

I added some comemnts to make it more explicit.

The check could be made in unit test but it means having mapping public.

I think I understand your idea - you want to check the newly added licenses (from another PR later).

I think we don't need to do this check every time we start Trivy.

The test can be done in a unit test, but this means that the mapping will be public.

You have added so many licenses, so I am not sure that many licenses will be added later.
Maybe a comment before mapping to the instructions for the new licenses will be enough. wdyt?

Especially with latest commit with the version regular expression, it's becoming quite difficult to make sure to only add standardized keys.

I added an InvalidMappingKeys used in test and which might also be used later if adding new mappings from CLI or other mean becomes possible.

DmitriyLewen · 2024-07-24T10:11:23Z

pkg/sbom/cyclonedx/marshal.go

@@ -288,7 +290,7 @@ func (*Marshaler) Licenses(licenses []string) *cdx.Licenses {
 	choices := lo.Map(licenses, func(license string, i int) cdx.LicenseChoice {
 		return cdx.LicenseChoice{
 			License: &cdx.License{
-				Name: license,
+				Name: NormalizeLicense(license),


Can we use this function here?

trivy/pkg/licensing/normalize.go

Lines 735 to 737 in 907683d

func Normalize(name string) string {

return NormalizeLicense(name).String()

}

I removed the CycloneDX normalization from this PR to reduce the scope.

From cycloneDX spec that would ideally mean having :

Id: "A valid SPDX license ID"

Name: "If SPDX does not define the license used, this field may be used to provide the license name",

integration/testdata/fluentd-multiple-lockfiles.cdx.json.golden

DmitriyLewen · 2024-07-24T10:36:50Z

pkg/fanal/test/integration/testdata/goldens/packages/amazon-2.json.golden

We do not check all fields for OS packages:

trivy/pkg/fanal/test/integration/library_test.go

Lines 321 to 324 in 051ac39

for i := 0; i < len(expectedPkgs); i++ {

require.Equal(t, expectedPkgs[i].Name, detail.Packages[i].Name, tc.name)

require.Equal(t, expectedPkgs[i].Version, detail.Packages[i].Version, tc.name)

}

Therefore, there is no point in updating the OS packages for the files pkg/fanal/test/integration/testdata/goldens/packages/*.json.golden.

I reverted those changes.

Sorry, I confused you a little.
containerd tests use full files.
I have updated these golden files in 39f796e.

Normalized SPDX license should be in Id field.

DmitriyLewen · 2024-07-25T03:24:40Z

pkg/licensing/normalize.go

+var mapping = make(map[string]expression.SimpleExpr)
+
+func addMap(name, key string, hasPlus bool) {
+	license := normalizeKeyAndSuffix(name)


I think I understand your idea - you want to check the newly added licenses (from another PR later).

I think we don't need to do this check every time we start Trivy.

The test can be done in a unit test, but this means that the mapping will be public.

You have added so many licenses, so I am not sure that many licenses will be added later.
Maybe a comment before mapping to the instructions for the new licenses will be enough. wdyt?

integration/testdata/fluentd-multiple-lockfiles.cdx.json.golden

DmitriyLewen · 2024-07-25T04:33:58Z

pkg/fanal/test/integration/testdata/goldens/packages/amazon-2.json.golden

Sorry, I confused you a little.
containerd tests use full files.
I have updated these golden files in 39f796e.

DmitriyLewen · 2024-07-25T04:34:45Z

pkg/fanal/test/integration/testdata/goldens/packages/vulnimage.json.golden

    },
    "Version": "1.6.3-r0",
    "Arch": "x86_64",
    "SrcName": "apr",
    "SrcVersion": "1.6.3-r0",
    "Licenses": [
-      "ASL2.0"
+      "Apache-2.02.0"


@pbaumard can you take a look?

Fixed in latest commit.

DmitriyLewen · 2024-07-31T08:43:18Z

pkg/licensing/normalize.go

+func addMap(name, key string, hasPlus bool) {
+	mapping[name] = expression.SimpleExpr{License: key, HasPlus: hasPlus}
+}


looks like we don't need this function anymore and we can just put everything in the map right away

Done in latest commit, even if I am not sure it is more readable that way.

DmitriyLewen · 2024-07-31T09:06:52Z

pkg/licensing/normalize.go

+var versionRegexpString = "([A-UW-Z)]{2,})( LICENSE)?\\s*[,(-]?\\s*(V|V\\.|VERSION|VERSION-|-)?\\s*([1-9](\\.\\d)*)[)]?"
+
+// case insensitive version match anywhere in string
+var versionRegexp = regexp.MustCompile("(?i)" + versionRegexpString)
+
+// version suffix match
+var versionSuffixRegexp = regexp.MustCompile(versionRegexpString + "$")


I'm not sure I understand where we need this.
Can you add an example in the comment?

I tried to avoid using regex if possible.
Maybe we can avoid using this regex here

The many version variations are in normalize_test.

Most of the mappings in current trivy version or in oss-review-toolkit/ort are because of slight variations in the way the version is declared in the license.

So this regexp allows to:

greatly limit the number of mappings

avoid missing version mappings

This regexp is strict by checking only version suffixes.

DmitriyLewen · 2024-07-31T09:08:16Z

pkg/licensing/normalize_test.go

@@ -8,6 +8,219 @@ import (
 	"github.com/aquasecurity/trivy/pkg/licensing"
 )

+func TestMap(t *testing.T) {
+	assert.Empty(t, licensing.InvalidMappingKeys((nil)))


Suggested change

assert.Empty(t, licensing.InvalidMappingKeys((nil)))

assert.Empty(t, licensing.InvalidMappingKeys(nil))

InvalidMappingKeys no more used in latest commit.

DmitriyLewen · 2024-07-31T09:11:17Z

pkg/fanal/analyzer/pkg/apk/apk_test.go

@@ -453,3 +450,23 @@ func TestParseApkInfo(t *testing.T) {
 		})
 	}
 }
+
+func TestParseLicense(t *testing.T) {


I think this is a redundant test.
We see this function working in TestParseApkInfo.

TestLaxSplitLicense was moved to normalize_test in latest commit.

DmitriyLewen · 2024-07-31T09:16:34Z

pkg/licensing/normalize.go

+func InvalidMappingKeys(licenseToNormalized map[string]expression.SimpleExpr) []string {
+	if licenseToNormalized == nil {
+		licenseToNormalized = mapping
+	}
+	var invalid []string
+	for key := range licenseToNormalized {
+		standardized := standardizeKeyAndSuffix(key)
+		if standardized.License != key {
+			invalid = append(invalid, key)
+		}
+	}
+	return invalid
+}


What if we move this function to the test.
To get mapping, we can create a function like GetBuiltinRules() for secrets.

normalize_test is now in package licensing to use internal mapping and functions.

…rmalize_test

DmitriyLewen

@pbaumard Thanks for work!
And sorry for delay.

LGTM.
Left 1 small comment.

DmitriyLewen · 2024-08-20T07:44:00Z

pkg/licensing/normalize_test.go

+			[]string{
+				"Apache+",
+			},
+			"Apache-2.0+",
+			"Apache-2.0",


Can you add field names for better readability in these tests?
e.g.:

Suggested change

[]string{

"Apache+",

},

"Apache-2.0+",

"Apache-2.0",

licenses: []string{

"Apache+",

},

normalized: "Apache-2.0+",

normalizedKey: "Apache-2.0",

Field names have been added in last commit.

…e-normalize

DmitriyLewen

Hello @pbaumard
Thanks for your work and sorry for delays.

@knqyf263 take a look, when you have time, please.

Signed-off-by: knqyf263 <[email protected]>

knqyf263

It's an awesome contribution! I planned to add this kind of improvement but didn't find the time. Thanks for your great work and patience.

* feat(vm): Support direct filesystem (aquasecurity#7058) Signed-off-by: yusuke.koyoshi <[email protected]> * feat(cli)!: delete deprecated SBOM flags (aquasecurity#7266) Signed-off-by: knqyf263 <[email protected]> * feat(vm): support the Ext2/Ext3 filesystems (aquasecurity#6983) * fix(plugin): do not call GitHub content API for releases and tags (aquasecurity#7274) Signed-off-by: knqyf263 <[email protected]> * fix(java): Return error when trying to find a remote pom to avoid segfault (aquasecurity#7275) Co-authored-by: DmitriyLewen <[email protected]> * fix(flag): incorrect behavior for deprected flag `--clear-cache` (aquasecurity#7281) * refactor(misconf): remove file filtering from parsers (aquasecurity#7289) Signed-off-by: nikpivkin <[email protected]> * feat(vuln): Add `--detection-priority` flag for accuracy tuning (aquasecurity#7288) Signed-off-by: knqyf263 <[email protected]> * docs: add auto-generated config (aquasecurity#7261) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * fix(terraform): add aws_region name to presets (aquasecurity#7184) * perf(misconf): do not convert contents of a YAML file to string (aquasecurity#7292) Signed-off-by: nikpivkin <[email protected]> * refactor(misconf): remove unused universal scanner (aquasecurity#7293) Signed-off-by: nikpivkin <[email protected]> * perf(misconf): use json.Valid to check validity of JSON (aquasecurity#7308) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): load only submodule if it is specified in source (aquasecurity#7112) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): support for policy and bucket grants (aquasecurity#7284) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): do not set default value for default_cache_behavior (aquasecurity#7234) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): iterator argument support for dynamic blocks (aquasecurity#7236) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: simar7 <[email protected]> * chore(deps): bump the common group across 1 directory with 7 updates (aquasecurity#7305) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * docs: update client/server docs for misconf and license scanning (aquasecurity#7277) Signed-off-by: nikpivkin <[email protected]> Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * docs: update links to packaging.python.org (aquasecurity#7318) Signed-off-by: nikpivkin <[email protected]> * perf(misconf): optimize work with context (aquasecurity#6968) Signed-off-by: nikpivkin <[email protected]> * refactor: replace ftypes.Gradle with packageurl.TypeGradle (aquasecurity#7323) Signed-off-by: nikpivkin <[email protected]> * docs: update air-gapped docs (aquasecurity#7160) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * docs(misconf): Update callsites to use correct naming (aquasecurity#7335) * chore(deps): bump the common group with 9 updates (aquasecurity#7333) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(misconf): change default TLS values for the Azure storage account (aquasecurity#7345) Signed-off-by: nikpivkin <[email protected]> * refactor(misconf): highlight only affected rows (aquasecurity#7310) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): wrap Azure PortRange in iac types (aquasecurity#7357) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): scanning support for YAML and JSON (aquasecurity#7311) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): variable support for Terraform Plan (aquasecurity#7228) Signed-off-by: nikpivkin <[email protected]> * fix: safely check if the directory exists (aquasecurity#7353) Signed-off-by: nikpivkin <[email protected]> * chore(deps): bump the aws group across 1 directory with 7 updates (aquasecurity#7358) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feat(server): add internal `--path-prefix` flag for client/server mode (aquasecurity#7321) Signed-off-by: knqyf263 <[email protected]> * chore(deps): bump trivy-checks (aquasecurity#7350) Signed-off-by: nikpivkin <[email protected]> * refactor(misconf): use slog (aquasecurity#7295) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): ignore duplicate checks (aquasecurity#7317) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): init frameworks before updating them (aquasecurity#7376) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): support deprecating for Go checks (aquasecurity#7377) Signed-off-by: nikpivkin <[email protected]> * feat(python): use minimum version for pip packages (aquasecurity#7348) * docs: add pkg flags to config file page (aquasecurity#7370) * feat(misconf): Add support for using spec from on-disk bundle (aquasecurity#7179) * fix(report): escape `Message` field in `asff.tpl` template (aquasecurity#7401) * fix(misconf): use module to log when metadata retrieval fails (aquasecurity#7405) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): support for ignore by nested attributes (aquasecurity#7205) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): do not filter Terraform plan JSON by name (aquasecurity#7406) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): port and protocol support for EC2 networks (aquasecurity#7146) Signed-off-by: nikpivkin <[email protected]> * chore: fix allow rule of ignoring test files to make it case insensitive (aquasecurity#7415) * fix(secret): use only line with secret for long secret lines (aquasecurity#7412) * chore: update CODEOWNERS (aquasecurity#7398) Signed-off-by: knqyf263 <[email protected]> * feat(server): Make Trivy Server Multiplexer Exported (aquasecurity#7389) * feat(report): export modified findings in JSON (aquasecurity#7383) Signed-off-by: knqyf263 <[email protected]> * fix(sbom): use `NOASSERTION` for licenses fields in SPDX formats (aquasecurity#7403) * fix(misconf): do not register Rego libs in checks registry (aquasecurity#7420) Signed-off-by: nikpivkin <[email protected]> * chore(deps): Bump trivy-checks (aquasecurity#7417) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: nikpivkin <[email protected]> * fix(misconf): do not recreate filesystem map (aquasecurity#7416) Signed-off-by: nikpivkin <[email protected]> * fix(secret): use `.eyJ` keyword for JWT secret (aquasecurity#7410) * fix(misconf): fix infer type for null value (aquasecurity#7424) Signed-off-by: nikpivkin <[email protected]> * fix(aws): handle ECR repositories in different regions (aquasecurity#6217) Signed-off-by: Kevin Conner <[email protected]> * fix: logger initialization before flags parsing (aquasecurity#7372) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * fix(nodejs): check all `importers` to detect dev deps from pnpm-lock.yaml file (aquasecurity#7387) * test: add integration plugin tests (aquasecurity#7299) * feat(sbom): set User-Agent header on requests to Rekor (aquasecurity#7396) Signed-off-by: Bob Callaway <[email protected]> * fix(helm): explicitly define `kind` and `apiVersion` of `volumeClaimTemplate` element (aquasecurity#7362) * chore(deps): Bump trivy-checks and pin OPA (aquasecurity#7427) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: nikpivkin <[email protected]> * feat(java): add `test` scope support for `pom.xml` files (aquasecurity#7414) * fix(license): add license handling to JUnit template (aquasecurity#7409) * feat(go): use `toolchain` as `stdlib` version for `go.mod` files (aquasecurity#7163) * release: v0.55.0 [main] (aquasecurity#7271) * fix(license): stop spliting a long license text (aquasecurity#7336) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * refactor(java): add error/statusCode for logs when we can't get pom.xml/maven-metadata.xml from remote repo (aquasecurity#7451) * chore(helm): bump up Trivy Helm chart (aquasecurity#7441) * chore(deps): bump the common group across 1 directory with 19 updates (aquasecurity#7436) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: knqyf263 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <[email protected]> * chore(deps): bump the aws group with 6 updates (aquasecurity#7468) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(oracle): Update EOL date for Oracle 7 (aquasecurity#7480) * fix(report): change a receiver of MarshalJSON (aquasecurity#7483) Signed-off-by: knqyf263 <[email protected]> * fix(report): fix error with unmarshal of `ExperimentalModifiedFindings` (aquasecurity#7463) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * docs(oci): Add a note About the expected Media Type for the Trivy-DB OCI Artifact (aquasecurity#7449) * feat(license): improve license normalization (aquasecurity#7131) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: DmitriyLewen <[email protected]> Co-authored-by: knqyf263 <[email protected]> * docs(db): add a manifest example (aquasecurity#7485) Signed-off-by: knqyf263 <[email protected]> * revert(java): stop supporting of `test` scope for `pom.xml` files (aquasecurity#7488) * docs: refine go docs (aquasecurity#7442) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * chore(vex): suppress openssl vulnerabilities (aquasecurity#7500) Signed-off-by: knqyf263 <[email protected]> * chore(deps): bump alpine from 3.20.0 to 3.20.3 (aquasecurity#7508) * chore(vex): add `CVE-2024-34155`, `CVE-2024-34156` and `CVE-2024-34158` in `trivy.openvex.json` (aquasecurity#7510) * fix(java): use `dependencyManagement` from root/child pom's for dependencies from parents (aquasecurity#7497) * refactor: split `.egg` and `packaging` analyzers (aquasecurity#7514) * feat(misconf): Register checks only when needed (aquasecurity#7435) * fix(misconf): Fix logging typo (aquasecurity#7473) * chore(deps): bump go-ebs-file (aquasecurity#7513) Signed-off-by: nikpivkin <[email protected]> * fix(sbom): parse type `framework` as `library` when unmarshalling `CycloneDX` files (aquasecurity#7527) * refactor(misconf): pass options to Rego scanner as is (aquasecurity#7529) Signed-off-by: nikpivkin <[email protected]> * fix(sbom): export bom-ref when converting a package to a component (aquasecurity#7340) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: amf <[email protected]> Co-authored-by: knqyf263 <[email protected]> * perf(misconf): use port ranges instead of enumeration (aquasecurity#7549) Signed-off-by: nikpivkin <[email protected]> * fix(misconf): Fixed scope for China Cloud (aquasecurity#7560) * docs(misconf): Add more info on how to use arbitrary JSON/YAML scan feat (aquasecurity#7458) * chore(deps): remove broken replaces for opa and discovery (aquasecurity#7600) * ci: cache test images for `integration`, `VM` and `module` tests (aquasecurity#7599) * ci: add `workflow_dispatch` trigger for test workflow. (aquasecurity#7606) * chore(deps): bump the common group across 1 directory with 20 updates (aquasecurity#7604) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: knqyf263 <[email protected]> * fix(db): check `DownloadedAt` for `trivy-java-db` (aquasecurity#7592) * fix: allow access to '..' in mapfs (aquasecurity#7575) Signed-off-by: nikpivkin <[email protected]> * test: use a local registry for remote scanning (aquasecurity#7607) Signed-off-by: knqyf263 <[email protected]> * fix(misconf): escape all special sequences (aquasecurity#7558) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): add ability to disable checks by ID (aquasecurity#7536) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: Simar <[email protected]> * feat(suse): added SUSE Linux Enterprise Micro support (aquasecurity#7294) Signed-off-by: Marcus Meissner <[email protected]> Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * fix(misconf): disable DS016 check for image history analyzer (aquasecurity#7540) Signed-off-by: nikpivkin <[email protected]> * ci: split `save` and `restore` cache actions (aquasecurity#7614) * refactor: fix auth error handling (aquasecurity#7615) Signed-off-by: knqyf263 <[email protected]> * feat(secret): enhance secret scanning for python binary files (aquasecurity#7223) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: knqyf263 <[email protected]> * feat(java): add empty versions if `pom.xml` dependency versions can't be detected (aquasecurity#7520) Co-authored-by: Teppei Fukuda <[email protected]> * test: use loaded image names (aquasecurity#7617) Signed-off-by: knqyf263 <[email protected]> * ci: don't use cache for `setup-go` (aquasecurity#7622) * feat: support multiple DB repositories for vulnerability and Java DB (aquasecurity#7605) Signed-off-by: nikpivkin <[email protected]> * feat(misconf): Support `--skip-*` for all included modules (aquasecurity#7579) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: nikpivkin <[email protected]> * chore: add prefixes to log messages (aquasecurity#7625) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: simar7 <[email protected]> * fix(misconf): Disable deprecated checks by default (aquasecurity#7632) * chore(deps): Bump trivy-checks to v1.1.0 (aquasecurity#7631) * fix(secret): change grafana token regex to find them without unquoted (aquasecurity#7627) * feat: support RPM archives (aquasecurity#7628) Signed-off-by: knqyf263 <[email protected]> * fix(misconf): not to warn about missing selectors of libraries (aquasecurity#7638) Signed-off-by: nikpivkin <[email protected]> * release: v0.56.0 [main] (aquasecurity#7447) * fix(db): fix javadb downloading error handling [backport: release/v0.56] (aquasecurity#7646) Signed-off-by: nikpivkin <[email protected]> Co-authored-by: Nikita Pivkin <[email protected]> * release: v0.56.1 [release/v0.56] (aquasecurity#7648) * fix(sbom): add options for DBs in private registries [backport: release/v0.56] (aquasecurity#7691) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: Teppei Fukuda <[email protected]> * fix(redhat): include arch in PURL qualifiers [backport: release/v0.56] (aquasecurity#7702) Signed-off-by: knqyf263 <[email protected]> Co-authored-by: Teppei Fukuda <[email protected]> * release: v0.56.2 [release/v0.56] (aquasecurity#7694) * Make liveness probe configurable (#3) --------- Signed-off-by: yusuke.koyoshi <[email protected]> Signed-off-by: knqyf263 <[email protected]> Signed-off-by: nikpivkin <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Kevin Conner <[email protected]> Signed-off-by: Bob Callaway <[email protected]> Signed-off-by: Marcus Meissner <[email protected]> Co-authored-by: yusuke-koyoshi <[email protected]> Co-authored-by: Teppei Fukuda <[email protected]> Co-authored-by: Aruneko <[email protected]> Co-authored-by: Colm O hEigeartaigh <[email protected]> Co-authored-by: DmitriyLewen <[email protected]> Co-authored-by: afdesk <[email protected]> Co-authored-by: Nikita Pivkin <[email protected]> Co-authored-by: Alberto Donato <[email protected]> Co-authored-by: simar7 <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Itay Shakury <[email protected]> Co-authored-by: DmitriyLewen <[email protected]> Co-authored-by: aasish-r <[email protected]> Co-authored-by: Ori <[email protected]> Co-authored-by: Kevin Conner <[email protected]> Co-authored-by: Bob Callaway <[email protected]> Co-authored-by: vhash <[email protected]> Co-authored-by: psibre <[email protected]> Co-authored-by: Aqua Security automated builds <[email protected]> Co-authored-by: s-reddy1498 <[email protected]> Co-authored-by: Squiddim <[email protected]> Co-authored-by: Pierre Baumard <[email protected]> Co-authored-by: Lior Kaplan <[email protected]> Co-authored-by: amf <[email protected]> Co-authored-by: bloomadcariad <[email protected]> Co-authored-by: Sylvain Baubeau <[email protected]> Co-authored-by: Simar <[email protected]> Co-authored-by: Marcus Meissner <[email protected]> Co-authored-by: Samuel Gaist <[email protected]>

Signed-off-by: knqyf263 <[email protected]> Co-authored-by: DmitriyLewen <[email protected]> Co-authored-by: knqyf263 <[email protected]>

(feat)Improve license normalization

44b0164

- Remove "THE " prefix - Remove "LICENSE " suffix - Fix lowercase key - Add BSD LICENSE 3 - Add ECLIPSE PUBLIC LICENSE 1.0 - Add MIT-0 - Add tests

pbaumard requested a review from knqyf263 as a code owner July 9, 2024 15:04

Fix indenting

5adc524

knqyf263 requested a review from DmitriyLewen July 9, 2024 17:41

DmitriyLewen reviewed Jul 10, 2024

View reviewed changes

Fix format and test

6b52af5

DmitriyLewen reviewed Jul 11, 2024

View reviewed changes

pkg/licensing/normalize_test.go Outdated Show resolved Hide resolved

mapping back to private and other suffixes ignore from SPDX

4dc5070

DmitriyLewen reviewed Jul 12, 2024

View reviewed changes

pkg/licensing/normalize.go Show resolved Hide resolved

pbaumard added 11 commits July 12, 2024 10:10

Do not change unmapped license name

60d29b0

Additional mappings from oss-review-toolkit and space normalization

ab62262

Fix double space

d0101fc

Add back GPL WITH AUTOCONF EXCEPTION

cd7d779

Normalize CycloneDX licenses and fix tests

aa7ee3c

Add MPL mappings

c21e723

Update golden test files

3055e10

Normalize suffixes using existing expression package

6c168f0

Fix copyright_test

6ec91a5

Add ValidateDocument in SPDX test

0f4c8cb

Fix linter

907683d

pbaumard requested a review from DmitriyLewen July 18, 2024 08:52

DmitriyLewen reviewed Jul 24, 2024

View reviewed changes

pbaumard added 5 commits July 24, 2024 15:21

Revert since only Name and Version are checked

5074f24

Revert cycloneDX normalization

8e6f9d3

Normalized SPDX license should be in Id field.

More tests

b8bb817

Comments

56093d9

Fix golden

85b8781

test(integration): update golden files

39f796e

DmitriyLewen reviewed Jul 25, 2024

View reviewed changes

Fix apk license parsing and use version regexp for license normalization

5f7cf6d

pbaumard requested a review from DmitriyLewen July 26, 2024 16:47

DmitriyLewen reviewed Jul 31, 2024

View reviewed changes

Check map in test; make map directly & move TestLaxSplitLicense to no…

aff6ebd

…rmalize_test

pbaumard requested a review from DmitriyLewen August 1, 2024 11:05

DmitriyLewen reviewed Aug 20, 2024

View reviewed changes

pbaumard and others added 3 commits September 2, 2024 17:01

Add field names in test

0134266

Merge branch 'aquasecurity:main' into feature/better-license-normalize

b85b781

Golden test update

8663dcd

pbaumard requested a review from DmitriyLewen September 3, 2024 09:10

Merge remote-tracking branch 'origin/main' into feature/better-licens…

932e937

…e-normalize

DmitriyLewen approved these changes Sep 11, 2024

View reviewed changes

test: add a file for private functions

f3f7c39

Signed-off-by: knqyf263 <[email protected]>

knqyf263 approved these changes Sep 11, 2024

View reviewed changes

knqyf263 enabled auto-merge September 11, 2024 06:30

knqyf263 added this pull request to the merge queue Sep 11, 2024

Merged via the queue into aquasecurity:main with commit 6472e3c Sep 11, 2024
12 checks passed

pbaumard deleted the feature/better-license-normalize branch September 11, 2024 13:06

aqua-bot mentioned this pull request Sep 11, 2024

release: v0.56.0 [main] #7447

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(license): improve license normalization #7131

feat(license): improve license normalization #7131

pbaumard commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024

DmitriyLewen left a comment

DmitriyLewen left a comment

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 25, 2024

pbaumard Jul 26, 2024

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 24, 2024

pbaumard Jul 24, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

DmitriyLewen Jul 25, 2024

pbaumard Jul 26, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen Jul 31, 2024

pbaumard Aug 1, 2024

DmitriyLewen left a comment

DmitriyLewen Aug 20, 2024

pbaumard Sep 2, 2024

DmitriyLewen left a comment

knqyf263 left a comment •

edited

Loading

	func Normalize(name string) string {
	return NormalizeLicense(name).String()
	}

	for i := 0; i < len(expectedPkgs); i++ {
	require.Equal(t, expectedPkgs[i].Name, detail.Packages[i].Name, tc.name)
	require.Equal(t, expectedPkgs[i].Version, detail.Packages[i].Version, tc.name)
	}

	assert.Empty(t, licensing.InvalidMappingKeys((nil)))
	assert.Empty(t, licensing.InvalidMappingKeys(nil))

feat(license): improve license normalization #7131

feat(license): improve license normalization #7131

Conversation

pbaumard commented Jul 9, 2024 • edited Loading

Description

Related issues

Checklist

CLAassistant commented Jul 9, 2024 • edited Loading

CLAassistant commented Jul 9, 2024

DmitriyLewen left a comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen left a comment

Choose a reason for hiding this comment

knqyf263 left a comment • edited Loading

Choose a reason for hiding this comment

pbaumard commented Jul 9, 2024 •

edited

Loading

CLAassistant commented Jul 9, 2024 •

edited

Loading

knqyf263 left a comment •

edited

Loading