Report Packages at the codebase-level

[JonoYang]

We should make the package consolidation logic from the consolidation plugin a default function of the Package scanning option. The consolidation plugin would then be focused on files that are not part of a package so we can perform logical groupings on them.

Some changes that would have to change on the Package model/Package scanning process:

• Remove root_path from Package. There is no universal root path for all Packages.
• Tag a resource if it is part of one or more Package during a Package scan. This would be similar to the consolidated_to field, where it would be a list of purls. Possible name for this field is for_packages.
• Return detected Packages in a new top level codebase attribute. This would be a set of all detected packages from a codebase.
• Add primary license expression/copyright/holders to Package models, which is populated from top-level key package files (manifests, etc.)
• Add secondary license expression/copyrights/holders to Package. This is populated from the detected license expressions/copyrights of Package resources, excluding top-level key files.

---

Here is an updated design:

The key elements are to:

• report packages as top-level. The data structure is the same as the one at the file level but will be the merged data from possibly several manifests and lock files.
• track which files are part of a given package instance

Package model updates

• Package model updates: Add new package_instance #2691
• Package model updates: Add package_manifest_paths attribute #2692
• Package model updates: Add new PackageManifest class #2747

Files model updates

• Files model updates: add for_packages attribute #2693
• Files model updates: Rename the current file-level packages to package_manifests. #2694

This could look this way:

packages (aka. package instances)
  package
    package_manifest_paths
    ... data...

files:
   package_manifests: [...] (formerly named packages)
   for_packages: [ list of 
     { package_url: pkg:foo/bar@12, package_instance: UUID}
   ]

For later:

• sub-packages/embedded could be 1) listed directly as packages of their own 2) related to their parent (or the parent related to them)
• we could also track the file_paths under each top level package instance
• get all packages to have actual files
• migrate system packages (Alpine, RPM, Debian, Windows) to new model

[pombredanne]

Something to consider if we were to track packages at the top level:

• should we track each instance of a package (and its files separately)
• what happens if different package instances have slightly metadata?

[pombredanne]

@JonoYang I reckon this was never merged. And we will need this but IMHO we should go one more step:

1. the things we detect today are incorrectly reported as packages when they really are package manifests
2. some packages may have multiple manifests (e.g. manifest proper and a lock file). For instance .gemspec/Gemfile/Gemfile.lock or package.json/package-lock.json/yarn.lock or setup.py/setup.cfg/requirements.txt ... etc.

Packages can be nested too:a package can have sub-packages or have multiple personalities such as a bower.json and a package.json, or node_modules nested under an npm or scancode-toolkit bundled wheels.

Therefore I think we should:

• rename the current file-level packages to package_manifests. There the data structure is that of a Package, but with no constraints about which field are mandatory.

• report packages only as top-level. The data structure is the same but is the merged data from possibly several manifests, and lock files.

• we want to track unique instances of each packages (say you have two identical wheels in the same code tree) and we could use simply a UUID for each instance. Files of a package would point to the purl + UUID, e.g. an instance of a package.

• sub-packages/embedded could be 1) listed directly as packages of their own 2) related to their parent (or the parent related to them)

[pombredanne]

After a chat with @tdruez , we should list right away the file_paths under each top level package instance as it becomes much clearer and simpler to handle in scancode.io.

[pombredanne]

From #1554

Packages often contain other nested packages from other origins. For instance an installed npm-based application will have a node_modules directory of the provisioned dependencies once an npm install command has been issued. The same would apply to Pypi packages in a virtualenv, Rubygems bundled in a vendor dir with bundler, nested Maven projects... etc.

Therefore it would be useful to attach the list of files that are part of a detected package in the packages list of returned results. In many cases this list of files is exactly the same as the descendants files of the package root directory ... but not always as explained above.

The usage will be to use these to perform a proper summarization of the license/origin/etc of the files that are part of a package at the package level. And this will support #377

[AyanSinhaMahapatra]

Also see improvements to this in #2843

[pombredanne]

This is mostly done but there are some smaller issues pending.

[pombredanne]

I consider this done now. @JonoYang @AyanSinhaMahapatra Thanks!