Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Conan (C/C++) conan.lock file support #1230

Merged
merged 12 commits into from
Sep 29, 2022
Merged

Add Conan (C/C++) conan.lock file support #1230

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
67d8f25
[CPP] add Conan conan.lock file support
hkwi Sep 28, 2022
96aac3a
update function based on linter error
spiffcs Sep 28, 2022
5ad3b0e
update struct to not be inline based on review
spiffcs Sep 28, 2022
e72dbe3
update conanLock with explicit json tags
spiffcs Sep 28, 2022
d44c78e
enhance Metadata with lockfile information
spiffcs Sep 28, 2022
010e58f
update ConanMetadata to better represent Ref
spiffcs Sep 28, 2022
fea2745
update conanlock to use pkg.ConanMetadata
spiffcs Sep 28, 2022
d3ed041
add new ConanLockMetadata
spiffcs Sep 28, 2022
c25aa38
generate new schema
spiffcs Sep 29, 2022
6401db3
update with correct metadata inclusion
spiffcs Sep 29, 2022
be74c52
remove duplicate type
spiffcs Sep 29, 2022
20560be
add PURL test back for ConanLockMetadata
spiffcs Sep 29, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions syft/pkg/cataloger/cpp/cataloger.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import (
func NewConanfileCataloger() *common.GenericCataloger {
globParsers := map[string]common.ParserFn{
"**/conanfile.txt": parseConanfile,
"**/conan.lock": parseConanlock,
}

return common.NewGenericCataloger(nil, globParsers, "conan-cataloger")
Expand Down
59 changes: 59 additions & 0 deletions syft/pkg/cataloger/cpp/parse_conanlock.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package cpp

import (
"encoding/json"
"io"
"strings"

"github.com/anchore/syft/syft/artifact"
"github.com/anchore/syft/syft/pkg"
"github.com/anchore/syft/syft/pkg/cataloger/common"
)

// integrity check
var _ common.ParserFn = parseConanlock

type conanLock struct {
GraphLock struct {
Nodes map[string]struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: this could be map[string]pkg.ConanMetadata instead of redefining these. That is, if the metadata is meant to raise up the raw packaging metadata, if we remove name/version then there would be no difference between this struct def and pkg.ConanMetadata.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could be wrong about this if options string should really be transformed into options []string (split by newline) or options map[string]string if split by newline and =. (same comment for similar fields)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll update options to be the map suggestion and look to make the other fields updated to match this suggestion

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For those following along here are all possible fields for the conan.lock structure
https://github.com/conan-io/conan/blob/develop/conans/model/graph_lock.py#L101-L276

Ref string `json:"ref"`
Options string `json:"options"`
Path string `json:"path"`
Context string `json:"context"`
} `json:"nodes"`
} `json:"graph_lock"`
Version string `json:"version"`
ProfileHost string `json:"profile_host"`
}

// parseConanlock is a parser function for conan.lock contents, returning all packages discovered.
func parseConanlock(_ string, reader io.Reader) ([]*pkg.Package, []artifact.Relationship, error) {
pkgs := []*pkg.Package{}
var cl conanLock
if err := json.NewDecoder(reader).Decode(&cl); err != nil {
return nil, nil, err
}
for _, node := range cl.GraphLock.Nodes {
if len(node.Ref) > 0 {
// ref: pkga/0.1@user/testing
splits := strings.Split(strings.Split(node.Ref, "@")[0], "/")
if len(splits) < 2 {
continue
}
pkgName, pkgVersion := splits[0], splits[1]
pkgs = append(pkgs, &pkg.Package{
Name: pkgName,
Version: pkgVersion,
Language: pkg.CPP,
Type: pkg.ConanPkg,
MetadataType: pkg.ConanaMetadataType,
Metadata: pkg.ConanMetadata{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

capturing the name and version isn't really necessary. There is one or two other catalogers that do this for implementation-specific limitations, however, in this case it's redundant.

That being said, there is other information in a conan.lock file that could be useful to capture (such as options). It's probably worth taking a look at what other information could be provided and adding that to the metadata.

If we don't want to add any more elements, my recommendation now would be to not have a metadata struct at all.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wagoodman I can add options path and context as metadata fields and remove name and version here

Then I'll update the conanfile to include nil for the metadata

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@wagoodman it also looks like name and version are tied to the method for generating a PURL for a given conan package. It might be useful to keep those fields so we can agnostically generate the PURL for either a lockfile or conanfile even if the information is redundant for the package struct:

https://github.com/anchore/syft/blob/main/syft/pkg/conan_metadata.go

Is there another space you would want this method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, we're reusing the same metadata for logically a different things. I don't think we should do this. That is, the Metadata is reserved for things that are specific from what this package was parsed from (I didn't see the reuse on the first review pass). We should probably be introducing a new metadata type to capture this information.

Name: pkgName,
Version: pkgVersion,
},
})
}
}

return pkgs, nil, nil
}
42 changes: 42 additions & 0 deletions syft/pkg/cataloger/cpp/parse_conanlock_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
package cpp

import (
"os"
"testing"

"github.com/go-test/deep"

"github.com/anchore/syft/syft/pkg"
)

func TestParseConanlock(t *testing.T) {
expected := []*pkg.Package{
{
Name: "zlib",
Version: "1.2.12",
Language: pkg.CPP,
Type: pkg.ConanPkg,
MetadataType: pkg.ConanaMetadataType,
Metadata: pkg.ConanMetadata{
Name: "zlib",
Version: "1.2.12",
},
},
}

fixture, err := os.Open("test-fixtures/conan.lock")
if err != nil {
t.Fatalf("failed to open fixture: %+v", err)
}

// TODO: no relationships are under test yet
actual, _, err := parseConanlock(fixture.Name(), fixture)
if err != nil {
t.Error(err)
}

differences := deep.Equal(expected, actual)
if differences != nil {
t.Errorf("returned package list differed from expectation: %+v", differences)
}
}
15 changes: 15 additions & 0 deletions syft/pkg/cataloger/cpp/test-fixtures/conan.lock
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
{
"graph_lock": {
"nodes": {
"0": {
"ref": "zlib/1.2.12",
"options": "fPIC=True\nshared=False",
"path": "all/conanfile.py",
"context": "host"
}
},
"revisions_enabled": false
},
"version": "0.4",
"profile_host": "[settings]\narch=x86_64\narch_build=x86_64\nbuild_type=Release\ncompiler=gcc\ncompiler.libcxx=libstdc++\ncompiler.version=9\nos=Linux\nos_build=Linux\n[options]\n[build_requires]\n[env]\n"
}