Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve SPDX decoding functionality #738

Merged
merged 48 commits into from
Feb 9, 2022
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
2948496
add spdx json decoder
wagoodman Jan 4, 2022
73233a0
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 6, 2022
84c7767
basic refactor existing code to use spdx go tools for decoding
kzantow Jan 7, 2022
dfefc96
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 7, 2022
3944425
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 11, 2022
0b9ae11
Fix some SPDX export issues
kzantow Jan 11, 2022
538d7cd
Fix some SPDX export issues
kzantow Jan 11, 2022
a783d5b
Merge branch 'spdx-export-issues' into add-spdx-decoder-take-2
kzantow Jan 11, 2022
1b648bb
Fix originator + tests
kzantow Jan 12, 2022
b01e97f
Merge branch 'spdx-export-issues' into add-spdx-decoder-take-2
kzantow Jan 12, 2022
164d57c
One more minor spec issue
kzantow Jan 12, 2022
37d703f
Merge branch 'spdx-export-issues' into add-spdx-decoder-take-2
kzantow Jan 12, 2022
e105e29
cleanjup
kzantow Jan 12, 2022
6d35d9c
Merge branch 'spdx-export-issues' into add-spdx-decoder-take-2
kzantow Jan 12, 2022
e9b8f20
fix syftjson decoding of relationships
kzantow Jan 13, 2022
5c32316
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 13, 2022
386931d
Merge branch 'syftjson-decode-relationships' into add-spdx-decoder-ta…
kzantow Jan 13, 2022
df23559
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 13, 2022
26366a5
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 20, 2022
38ecea7
fix syftjson decoding of relationships
kzantow Jan 26, 2022
2125e54
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 26, 2022
d963fa4
lint
kzantow Jan 26, 2022
dc79ced
lint
kzantow Jan 26, 2022
582337b
lint
kzantow Jan 26, 2022
6c70f6b
remove accidental commit of sbom input
kzantow Jan 26, 2022
a116e5c
failing tests
kzantow Jan 26, 2022
8b01556
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 27, 2022
5f2b625
refactored
kzantow Jan 27, 2022
2db82c2
handle purl decoding and add basic test
kzantow Jan 27, 2022
ea73704
fail
kzantow Jan 27, 2022
cc18a37
cleanup
kzantow Jan 27, 2022
639e115
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Jan 28, 2022
ec41311
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Feb 2, 2022
b32c157
address some PR feedback / cleanup
kzantow Feb 2, 2022
1900d4e
more PR feedback
kzantow Feb 2, 2022
f519c94
more PR feedback
kzantow Feb 2, 2022
518139a
lint
kzantow Feb 2, 2022
0d60c50
remove setting any locations for spdx decoding
kzantow Feb 7, 2022
e82b487
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Feb 7, 2022
49958e4
update spdx license list
kzantow Feb 7, 2022
be763e7
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Feb 7, 2022
89a5df7
hopefully add proper file handling
kzantow Feb 7, 2022
f4b9794
lint
kzantow Feb 7, 2022
69e5bc9
imports
kzantow Feb 7, 2022
4ebcdcb
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Feb 7, 2022
ff1d6e7
Merge remote-tracking branch 'upstream/main' into add-spdx-decoder-ta…
kzantow Feb 8, 2022
9a216fc
oops, leftover debugging a panic
kzantow Feb 8, 2022
b9a93ea
stray printf
kzantow Feb 8, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ require (
github.com/scylladb/go-set v1.0.2
github.com/sergi/go-diff v1.1.0
github.com/sirupsen/logrus v1.8.1
github.com/spdx/tools-golang v0.1.0
github.com/spdx/tools-golang v0.2.0
github.com/spf13/afero v1.6.0
github.com/spf13/cobra v1.2.1
github.com/spf13/pflag v1.0.5
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -759,8 +759,8 @@ github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
github.com/spdx/gordf v0.0.0-20201111095634-7098f93598fb/go.mod h1:uKWaldnbMnjsSAXRurWqqrdyZen1R7kxl8TkmWk2OyM=
github.com/spdx/tools-golang v0.1.0 h1:iDMNEPqQk6CdiDj6eWDIDw85j0wQ3IR3pH9p0X05TSQ=
github.com/spdx/tools-golang v0.1.0/go.mod h1:RO4Y3IFROJnz+43JKm1YOrbtgQNljW4gAPpA/sY2eqo=
github.com/spdx/tools-golang v0.2.0 h1:KBNcw7xvVycRWeCWZK/5xQJA+plymW1+rTCs8ekJDro=
github.com/spdx/tools-golang v0.2.0/go.mod h1:RO4Y3IFROJnz+43JKm1YOrbtgQNljW4gAPpA/sY2eqo=
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
github.com/spf13/afero v1.6.0 h1:xoax2sJ2DT8S8xA2paPFjDCScCNeWsg75VG0DLRreiY=
Expand Down
5 changes: 4 additions & 1 deletion internal/formats/common/spdxhelpers/download_location.go
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@ package spdxhelpers

import "github.com/anchore/syft/syft/pkg"

const NONE = "NONE"
const NOASSERTION = "NOASSERTION"

func DownloadLocation(p pkg.Package) string {
// 3.7: Package Download Location
// Cardinality: mandatory, one
Expand All @@ -19,5 +22,5 @@ func DownloadLocation(p pkg.Package) string {
return NoneIfEmpty(metadata.URL)
}
}
return "NOASSERTION"
return NOASSERTION
}
4 changes: 2 additions & 2 deletions internal/formats/common/spdxhelpers/download_location_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func Test_DownloadLocation(t *testing.T) {
{
name: "no metadata",
input: pkg.Package{},
expected: "NOASSERTION",
expected: NOASSERTION,
},
{
name: "from apk",
Expand All @@ -43,7 +43,7 @@ func Test_DownloadLocation(t *testing.T) {
URL: "",
},
},
expected: "NONE",
expected: NONE,
},
}
for _, test := range tests {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
package model
package spdxhelpers

type ReferenceCategory string

Expand Down
41 changes: 8 additions & 33 deletions internal/formats/common/spdxhelpers/external_refs.go
Original file line number Diff line number Diff line change
@@ -1,51 +1,26 @@
package spdxhelpers

import (
"github.com/anchore/syft/internal/formats/spdx22json/model"
"github.com/anchore/syft/internal/log"
"github.com/anchore/syft/syft/pkg"
)

func ExternalRefs(p pkg.Package) (externalRefs []model.ExternalRef) {
externalRefs = make([]model.ExternalRef, 0)
func ExternalRefs(p pkg.Package) (externalRefs []ExternalRef) {
externalRefs = make([]ExternalRef, 0)

for _, c := range p.CPEs {
externalRefs = append(externalRefs, model.ExternalRef{
ReferenceCategory: model.SecurityReferenceCategory,
externalRefs = append(externalRefs, ExternalRef{
ReferenceCategory: SecurityReferenceCategory,
ReferenceLocator: pkg.CPEString(c),
ReferenceType: model.Cpe23ExternalRefType,
ReferenceType: Cpe23ExternalRefType,
})
}

if p.PURL != "" {
externalRefs = append(externalRefs, model.ExternalRef{
ReferenceCategory: model.PackageManagerReferenceCategory,
externalRefs = append(externalRefs, ExternalRef{
ReferenceCategory: PackageManagerReferenceCategory,
ReferenceLocator: p.PURL,
ReferenceType: model.PurlExternalRefType,
ReferenceType: PurlExternalRefType,
})
}
return externalRefs
}

func ExtractPURL(refs []model.ExternalRef) string {
for _, r := range refs {
if r.ReferenceType == model.PurlExternalRefType {
return r.ReferenceLocator
}
}
return ""
}

func ExtractCPEs(refs []model.ExternalRef) (cpes []pkg.CPE) {
for _, r := range refs {
if r.ReferenceType == model.Cpe23ExternalRefType {
cpe, err := pkg.NewCPE(r.ReferenceLocator)
if err != nil {
log.Warnf("unable to extract SPDX CPE=%q: %+v", r.ReferenceLocator, err)
continue
}
cpes = append(cpes, cpe)
}
}
return cpes
}
13 changes: 6 additions & 7 deletions internal/formats/common/spdxhelpers/external_refs_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ package spdxhelpers
import (
"testing"

"github.com/anchore/syft/internal/formats/spdx22json/model"
"github.com/anchore/syft/syft/pkg"
"github.com/stretchr/testify/assert"
)
Expand All @@ -13,7 +12,7 @@ func Test_ExternalRefs(t *testing.T) {
tests := []struct {
name string
input pkg.Package
expected []model.ExternalRef
expected []ExternalRef
}{
{
name: "cpe + purl",
Expand All @@ -23,16 +22,16 @@ func Test_ExternalRefs(t *testing.T) {
},
PURL: "a-purl",
},
expected: []model.ExternalRef{
expected: []ExternalRef{
{
ReferenceCategory: model.SecurityReferenceCategory,
ReferenceCategory: SecurityReferenceCategory,
ReferenceLocator: pkg.CPEString(testCPE),
ReferenceType: model.Cpe23ExternalRefType,
ReferenceType: Cpe23ExternalRefType,
},
{
ReferenceCategory: model.PackageManagerReferenceCategory,
ReferenceCategory: PackageManagerReferenceCategory,
ReferenceLocator: "a-purl",
ReferenceType: model.PurlExternalRefType,
ReferenceType: PurlExternalRefType,
},
},
},
Expand Down
17 changes: 17 additions & 0 deletions internal/formats/common/spdxhelpers/file_type.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
package spdxhelpers

type FileType string

const (
DocumentationFileType FileType = "DOCUMENTATION" // if the file serves as documentation
ImageFileType FileType = "IMAGE" // if the file is associated with a picture image file (MIME type of image/*, e.g., .jpg, .gif)
VideoFileType FileType = "VIDEO" // if the file is associated with a video file type (MIME type of video/*)
ArchiveFileType FileType = "ARCHIVE" // if the file represents an archive (.tar, .jar, etc.)
SpdxFileType FileType = "SPDX" // if the file is an SPDX document
ApplicationFileType FileType = "APPLICATION" // if the file is associated with a specific application type (MIME type of application/*)
SourceFileType FileType = "SOURCE" // if the file is human readable source code (.c, .html, etc.)
BinaryFileType FileType = "BINARY" // if the file is a compiled object, target image or binary executable (.o, .a, etc.)
TextFileType FileType = "TEXT" // if the file is human readable text file (MIME type of text/*)
AudioFileType FileType = "AUDIO" // if the file is associated with an audio file (MIME type of audio/* , e.g. .mp3)
OtherFileType FileType = "OTHER" // if the file doesn't fit into the above categories (generated artifacts, data files, etc.)
)
4 changes: 2 additions & 2 deletions internal/formats/common/spdxhelpers/license.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ func License(p pkg.Package) string {
// (iii) the SPDX file creator has intentionally provided no information (no meaning should be implied by doing so).

if len(p.Licenses) == 0 {
return "NONE"
return NONE
}

// take all licenses and assume an AND expression; for information about license expressions see https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/
Expand All @@ -30,7 +30,7 @@ func License(p pkg.Package) string {
}

if len(parsedLicenses) == 0 {
return "NOASSERTION"
return NOASSERTION
}

return strings.Join(parsedLicenses, " AND ")
Expand Down
4 changes: 2 additions & 2 deletions internal/formats/common/spdxhelpers/license_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ func Test_License(t *testing.T) {
{
name: "no licenses",
input: pkg.Package{},
expected: "NONE",
expected: NONE,
},
{
name: "no SPDX licenses",
Expand All @@ -25,7 +25,7 @@ func Test_License(t *testing.T) {
"made-up",
},
},
expected: "NOASSERTION",
expected: NOASSERTION,
},
{
name: "with SPDX license",
Expand Down
2 changes: 1 addition & 1 deletion internal/formats/common/spdxhelpers/none_if_empty.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (

func NoneIfEmpty(value string) string {
if strings.TrimSpace(value) == "" {
return "NONE"
return NONE
}
return value
}
6 changes: 3 additions & 3 deletions internal/formats/common/spdxhelpers/none_if_empty_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -20,17 +20,17 @@ func Test_noneIfEmpty(t *testing.T) {
{
name: "empty",
value: "",
expected: "NONE",
expected: NONE,
},
{
name: "space",
value: " ",
expected: "NONE",
expected: NONE,
},
{
name: "tab",
value: "\t",
expected: "NONE",
expected: NONE,
},
}
for _, test := range tests {
Expand Down
Loading