Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add identifiable field to source object #1243

Merged
merged 5 commits into from
Oct 5, 2022
Merged

Conversation

spiffcs
Copy link
Contributor

@spiffcs spiffcs commented Oct 4, 2022

Allow source.Source struct to set reproducible id for the following scheme:

  • DirectoryScheme
  • FileScheme
  • ImageScheme
  • UnknownScheme

This ID is calculated in the following ways:

Followups coming in separate smaller PR:

  • Add ID field to source and generate a relationship to packages
  • Update SPDX format to reflect this relationship of source --> package via OCI PURL

Signed-off-by: Christopher Phillips [email protected]

Allow source.Source struct to set reproducable id for different scheme.
This ID is calcuated either as a digest from the given directory or file
path. If the scheme is detected to be an ImageScheme then the ID is
calculated as a ChainID: https://github.com/opencontainers/image-spec/blob/main/config.md#layer-chainid

Signed-off-by: Christopher Phillips <[email protected]>
@github-actions
Copy link

github-actions bot commented Oct 4, 2022

Benchmark Test Results

Benchmark results from the latest changes vs base branch
name                                                       old time/op    new time/op    delta
ImagePackageCatalogers/alpmdb-cataloger-2                    12.5ms ±16%    15.1ms ±14%  +20.61%  (p=0.032 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2              1.31ms ± 1%    1.65ms ± 9%  +25.67%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2            3.33ms ± 0%    3.85ms ± 3%  +15.55%  (p=0.008 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2    1.08ms ± 0%    1.31ms ± 1%  +21.42%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         755µs ± 0%     908µs ± 2%  +20.20%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     892µs ± 1%    1110µs ± 3%  +24.35%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpm-db-cataloger-2                    1.29ms ± 1%    1.59ms ± 4%  +23.09%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      14.6ms ± 1%    17.3ms ± 4%  +18.17%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.26ms ± 1%    1.46ms ± 7%  +15.90%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2          2.22µs ± 1%    2.90µs ± 4%  +30.65%  (p=0.008 n=5+5)
ImagePackageCatalogers/dotnet-deps-cataloger-2               1.38ms ± 0%    1.70ms ± 2%  +23.50%  (p=0.008 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    706µs ± 0%     862µs ± 3%  +22.13%  (p=0.008 n=5+5)

name                                                       old alloc/op   new alloc/op   delta
ImagePackageCatalogers/alpmdb-cataloger-2                    5.26MB ± 0%    5.26MB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               202kB ± 0%     202kB ± 0%     ~     (p=0.802 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             945kB ± 0%     946kB ± 0%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     214kB ± 0%     214kB ± 0%     ~     (p=0.310 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         158kB ± 0%     158kB ± 0%   +0.07%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                     203kB ± 0%     203kB ± 0%     ~     (p=0.556 n=4+5)
ImagePackageCatalogers/rpm-db-cataloger-2                     302kB ± 0%     302kB ± 0%   -0.10%  (p=0.032 n=5+5)
ImagePackageCatalogers/java-cataloger-2                      3.44MB ± 0%    3.44MB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                     1.25MB ± 0%    1.25MB ± 0%     ~     (p=0.690 n=5+5)
ImagePackageCatalogers/go-module-binary-cataloger-2            672B ± 0%      672B ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                369kB ± 0%     369kB ± 0%     ~     (p=0.222 n=5+5)
ImagePackageCatalogers/portage-cataloger-2                    136kB ± 0%     136kB ± 0%   +0.04%  (p=0.032 n=5+5)

name                                                       old allocs/op  new allocs/op  delta
ImagePackageCatalogers/alpmdb-cataloger-2                     85.7k ± 0%     85.7k ± 0%     ~     (p=0.881 n=5+5)
ImagePackageCatalogers/ruby-gemspec-cataloger-2               4.25k ± 0%     4.25k ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2             16.6k ± 0%     16.6k ± 0%     ~     (p=0.730 n=5+5)
ImagePackageCatalogers/php-composer-installed-cataloger-2     5.53k ± 0%     5.53k ± 0%     ~     (p=0.421 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2         3.32k ± 0%     3.32k ± 0%     ~     (p=0.333 n=5+4)
ImagePackageCatalogers/dpkgdb-cataloger-2                     4.60k ± 0%     4.60k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpm-db-cataloger-2                     8.13k ± 0%     8.13k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                       57.5k ± 0%     57.5k ± 0%     ~     (p=0.952 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                      5.43k ± 0%     5.43k ± 0%     ~     (p=0.968 n=5+4)
ImagePackageCatalogers/go-module-binary-cataloger-2            15.0 ± 0%      15.0 ± 0%     ~     (all equal)
ImagePackageCatalogers/dotnet-deps-cataloger-2                7.27k ± 0%     7.27k ± 0%     ~     (all equal)
ImagePackageCatalogers/portage-cataloger-2                    3.59k ± 0%     3.59k ± 0%     ~     (all equal)

func (s *Source) SetID() {
if s.Metadata.Scheme != ImageScheme {
// How do we generate ID for non-image sources?
s.id = digest.FromString(s.Metadata.Path).String()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No path digest here: I think we want what's seen here
https://github.com/anchore/syft/pull/1218/files#diff-b79b344d258161c0ca7b6347af27d5ed643fdb01f4fbe654a69a688efe6a97cf

Getting a sum from the file itself or the metadata from the dir

Signed-off-by: Christopher Phillips <[email protected]>
@spiffcs spiffcs marked this pull request as ready for review October 5, 2022 15:05
[optional footer(s)]

Signed-off-by: Christopher Phillips <[email protected]>
Copy link
Contributor

@kzantow kzantow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

return s.id
}

func (s *Source) SetID() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this PR include a call to this function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ Second PR is going to do that so we can link it into relationships - Just wanted to keep this PR small, but I can add the call so we're setting it on construction

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking again, there also isn't ONE great place to put the single call since we have functions like New, NewFromImage NewFromDirectory NewFromFile.

Let me see all the places it needs to be included so we have full coverage of all the source creation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could just have it get set on the first call to ID if s.id = ""

@spiffcs spiffcs merged commit 7d2fe9d into main Oct 5, 2022
@spiffcs spiffcs deleted the 1241-syft-json-updates branch October 5, 2022 18:01
spiffcs added a commit to cpendery/syft that referenced this pull request Oct 11, 2022
* main:
  refactor: Remove experimental Anchore Enterprise upload functionality (anchore#1257)
  Update syft bootstrap tools to latest versions. (anchore#1254)
  Update Stereoscope to d24c9d626b33fa720210b007a20767801827b532 (anchore#1253)
  Update syft bootstrap tools to latest versions. (anchore#1244)
  fix apkdb checksum representation (anchore#1247)
  feat: add identifiable field to source object (anchore#1243)
  feat: attest support for Singularity images (anchore#1201)
  Update syft bootstrap tools to latest versions. (anchore#1239)
  Update Stereoscope to 1b1b744a919964f38d14e1416fb3f25221b761ce (anchore#1240)
  fix: Follow symlinks when searching for globs in all-layers scope (anchore#1221)
spiffcs added a commit that referenced this pull request Oct 13, 2022
* main: (45 commits)
  feat: add RelationshipsBySourceOwnership to syft json output (#1248)
  fix: reset merged package into map; (#1258)
  refactor: Remove experimental Anchore Enterprise upload functionality (#1257)
  Update syft bootstrap tools to latest versions. (#1254)
  Update Stereoscope to d24c9d626b33fa720210b007a20767801827b532 (#1253)
  Update syft bootstrap tools to latest versions. (#1244)
  fix apkdb checksum representation (#1247)
  feat: add identifiable field to source object (#1243)
  feat: attest support for Singularity images (#1201)
  Update syft bootstrap tools to latest versions. (#1239)
  Update Stereoscope to 1b1b744a919964f38d14e1416fb3f25221b761ce (#1240)
  fix: Follow symlinks when searching for globs in all-layers scope (#1221)
  update requires to use list; remove field (#1234)
  Add Conan (C/C++) conan.lock file support (#1230)
  add sequence diagrams and flesh out TODO notes (#1233)
  Do not fail if unable to parse `.rpm` file (#1232)
  fix: support exclude patterns on Windows (#1228)
  Update syft bootstrap tools to latest versions. (#1225)
  Update Stereoscope to 56552770e555d764ea72b99d3c810326b27ead4a (#1224)
  Update syft bootstrap tools to latest versions. (#1223)
  ...

Signed-off-by: Christopher Phillips <[email protected]>
aiwantaozi pushed a commit to aiwantaozi/syft that referenced this pull request Oct 20, 2022
spiffcs added a commit that referenced this pull request Oct 21, 2022
spiffcs added a commit that referenced this pull request Oct 21, 2022
GijsCalis pushed a commit to GijsCalis/syft that referenced this pull request Feb 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants