Expanded License Details in DejaCode

[DennisClark]

Expanded License Details in DejaCode
Draft status. Ready for comments, suggestions, and questions.
Proposed changes to the Package and Component models and UI are ready for review.
Proposed changes to the Product Relation models and UI are not yet defined, but will be available soon.
Potential changes to the Subcomponent model are still in the early stage of concept development.
Design details for Issue #63
Note: Enhancements to the package dependency tree model, depending on the completion of that initiative in SCIO and PurlDB, are also planned. It may be preferable to handle those enhancements separately to avoid an overly complicated upgrade.
Background
Objective: provide more clarity for "Declared License" vs "Concluded License" .
Benefit: support the completeness of an SBOM.
Create an additional declared_license field on Package. When a package scan is completed update both the current assigned_license field and this new declared_license field with the same values. The intention is to retain the declared_license as an historical record, so that the assigned_license field essentially becomes the "concluded license" (we can change the help text on that field).
Store the additional licenses (aka "detected licenses" or "other licenses") from the scan results on the package model as well. This will support deeper analysis and reporting, enabling users to comment on why specific additional licenses impact or do not impact the licensing terms as the package is expected to be used in an organization.
Commonly Used License Terms
We want to standardize on the following license terms, or at least reference these terms, to be in sync with the open source community:
declared license: a license expression derived from statements in the key files of a software project, such as the NOTICE, COPYING, README, and LICENSE files.
detected licenses: license expressions derived from clues in the various files of a software project, which are very often third-party software used by the project, or test, sample and documentation files.
concluded license: a license expression curated from the declared license, where the curator has performed analysis to clarify or correct the declared license, possibly including one or more detected licenses in the license expression. In DejaCode, this is the license expression assigned to a Package or Component.
effective license: a license expression curated in the context of the usage of a Package or Component in a specific Product context, which may assert a license choice when that is an option. In DejaCode this is a Product Relation (Inventory Item) license expression.
DejaCode Package/Component models:
license_expression
ScanCode.io DiscoveredPackage and PurlDB Package model:
declared_license_expression
declared_license_expression_spdx
license_detections
other_license_expression
other_license_expression_spdx
other_license_detections
extracted_license_statement
Current Model Notes:
ScanCode.io and PurlDB share the same license-related fields. While adding new fields to DejaCode, let's keep naming consistency to ease the import of data from SCIO and PurlDB.
The declared_license_expression value is the one put in the DejaCode.license_expression during import. That field is currently a mix of data that can be "declared" or "concluded"

Package and Component Model and UI Updates
Expand the help text currently associated with the DejaCode.license_expression :
The License Expression assigned to a DejaCode Package or Component is an editable value equivalent to a “concluded license” as determined by a curator who has performed analysis to clarify or correct the declared license expression, which may have been assigned automatically (from a scan or an associated package definition) when the Package or Component was originally created. A license expression defines the relationship of one or more licenses to a software object. More than one applicable license can be expressed as "license-key-a AND license-key-b". A choice of applicable licenses can be expressed as "license-key-a OR license-key-b", and you can indicate the primary (preferred) license by placing it first, on the left-hand side of the OR relationship. The relationship words (OR, AND) can be combined as needed, and the use of parentheses can be applied to clarify the meaning; for example "((license-key-a AND license-key-b) OR (license-key-c))". An exception to a license can be expressed as “license-key WITH license-exception-key".
New Field Definitions
Starting with DejaCode Package and Component, since they are context-independent, let's add the additional license fields currently defined in PurlDB, using the exact same formats as the PurlDB and SCIO for each field:
declared_license_expression
Label: Declared license expression
Text: A license expression derived from statements in the key files of a software project, such as the NOTICE, COPYING, README, and LICENSE files.
declared_license_expression_spdx
Label: Declared license expression SPDX
Text: A declared license expression that uses the license identifiers defined by SPDX, as well as the “Licenseref” syntax for licenses not on the SPDX list.
license_detections
Label: License detections
Text: A list of specific license identifiers derived from statements in the key files of a software project, such as the NOTICE, COPYING, README, and LICENSE files.
other_license_expression
Label: Other license expression
Text: A license expression derived from detected licenses in the non-key files of a software project, which are very often third-party software used by the project, or test, sample and documentation files.
other_license_expression_spdx
Label: Other license expression SPDX
Text: A license expression derived from detected licenses in the non-key files of a software project, using the license identifiers defined by SPDX, as well as the “Licenseref” syntax for licenses not on the SPDX list.
other_license_detections
Label: Other license detections
Text: A list of specific license identifiers derived from the non-key files of a software project, which are very often third-party software used by the project, or test, sample and documentation files.
extracted_license_statement
Label: Extracted_license_statement
Text: The actual text extracted from a software project that supports the license detection process.
Package “Others”
In DejaCode the Package model contains four fields grouped together in the UI as “Others”. Conceptually, there is some overlap between these fields and the new, more precise, proposed fields, but it may be best to retain them as they are, even though they are scarcely populated in the current reference data:
Declared license (this is a completely free-form text string containing non-validated license identifiers, and is NOT the same as the new “Declared license expression” field). Used by 90 packages in current Reference Data.
Parties (a blob). Used by 172 packages in current Reference Data.
Datasource id (a free-form text string). Not populated at all in current Reference Data. Consider deprecating.
File references (a blob). Not populated at all in current Reference Data. Consider deprecating.
Package/Component User Detail Views
In DejaCode the Package and Component user detail views share the same format on the License tab, and the following proposed changes apply to both objects. Each of them displays a
“License expression” : Change label to “Concluded license expression” and change the help text as described above.
“Reference notes” (displayed when not empty): No change.
Add the following fields below them, with labels and text as described above:
Declared license expression
Declared license expression SPDX
License detections
Other license expression
Other license expression SPDX
Other license detections
Extracted license statement
Package/Component User Detail Edit Forms
In DejaCode the Package and Component user detail edit forms share (generally) the same format. The label for “License expression” should be changed to “Concluded license expression” and the help text should also be expanded as described above.
Package/Component User List Views
In DejaCode the Package and Component user list views share (generally) the same format. The column label for “License” should be changed to “Concluded license” and the help text should also be expanded as described above.
Package/Component Admin Detail Edit Form
In DejaCode the Package and Component admin detail edit forms require these modifications:
Change the label for “License expression” to “Concluded license expression” and expand the help text. Add the following fields below this one:
Declared license expression
Declared license expression SPDX (not editable, always automatically generated from Declared license expression)
License detections
Other license expression
Other license expression SPDX (not editable, always automatically generated from Other license expression)
Other license detections
Extracted license statement
Package/Component Reporting/Scan/Import/API
Package scan: Enhance the process to populate the new fields as available from scan results.
In particular, note that both the License expression (“Concluded license expression”) and Declared license expression should be automatically populated with the same values as the result of a Package scan.
Package and Component Reporting: All the new fields should be available in the DejaCode Query and Column Template for their respective objects.
Package and Component API: All the new fields should be available in the DejaCode API for their respective objects.
Package and Component Import: All the new fields, except the auto-generated SPDX fields, should be available in the DejaCode Import for their respective objects.

Product Model and UI Updates
To be provided.

Epilogue