feat(c): add license support for conan lock files #6329

DmitriyLewen · 2024-03-15T10:07:19Z

Description

Add license support for conan dependencies.
Licenses are looked for in conanfile.py files from the conan cache directory.

Related issues

Close Conan license support #5944

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

…license

knqyf263 · 2024-04-22T10:29:52Z

pkg/fanal/analyzer/language/c/conan/conan.go

+				}
+			} else if strings.HasPrefix(line, "license") { // cf. https://docs.conan.io/1/reference/conanfile/attributes.html#license
+				// trim extra characters - e.g. `license = "Apache-2.0"` -> `Apache-2.0`
+				license = strings.TrimSuffix(strings.TrimPrefix(line, `license = "`), `"`)


I found this example in the doc.

from conan import ConanFile class MyAndroidNDKRecipe(ConanFile): name="my_android_ndk" version="1.0" def package_info(self): self.conf_info.define("tools.android:ndk_path", "bar")

We probably need a regexp like name\s*=\s*"([^"]+)". I want to avoid using regular expressions wherever possible, though.

In addition, if a conanfile.py is small enough, applying regexp to the file content might be simpler and faster.

var ( namePattern = regexp.MustCompile(`^\s*name\s*=\s*"(?P<name>[^"]+)"$`) licensePattern = regexp.MustCompile(`^\s*license\s*=\s*"(?P<license>[^"]+)"$`) ) ... content, err := io.ReadAll(r) if err != nil {...} // Find name if match := namePattern.FindSubmatch(content); len(match) > 0 { name = string(match[namePattern.SubexpIndex("name")]) } // Find license if match := licensePattern.FindSubmatch(content); if len(match) > 0 { license = string(match[licensePattern.SubexpIndex("version")]) }

I am not 100% sure which is faster as I have not compared it with processing one line at a time. I was just thinking out loud, and it's not a performance critical process. You can decide.

name="my_android_ndk"

It looks as bug, but we can also parse this case.

We probably need a regexp like name\s*=\s*"([^"]+)". I want to avoid using regular expressions wherever possible, though.

I also don't trust regexp and try not to use them.
I think we can just remove all the spaces.

I am not 100% sure which is faster as I have not compared it with processing one line at a time.

In the files that I checked, name and license fields are placed in first 20-30 lines.
So I think we won't gain much time by doing this.

if a conanfile.py is small enough, applying regexp to the file content might be simpler and faster.

675 lines - https://github.com/aquasecurity/trivy/blob/61af875ec3120ca1fcbae2a7e684173b40ca71b5/pkg/fanal/analyzer/language/c/conan/testdata/cacheDir/.conan/data/openssl/3.0.5/_/_/export/conanfile.py

I added removing extra spaces - 89003d6

It looks as bug, but we can also parse this case.

Why is it a bug? It does not break the Python semantics.

I haven't seen such cases. Also, all test cases use the format name = "foo". Therefore, I assumed that this was a bug or maybe just a typo in the documentation.

In any case, this is easy to solve.

Conanfile is written by hand, so any format is possible, isn't it? I think there is a customary preferred format, but it is legitimate.

I think there is a customary preferred format, but it is legitimate.

You are right 👍

pkg/fanal/analyzer/language/c/conan/conan.go

knqyf263 · 2024-04-23T09:26:27Z

pkg/fanal/analyzer/language/c/conan/conan.go

+				}
+			} else if strings.HasPrefix(line, "license") { // cf. https://docs.conan.io/1/reference/conanfile/attributes.html#license
+				// remove spaces before and after `=` (if used). e.g. `license = "Apache-2.0"` -> `license="Apache-2.0"`
+				license = strings.ReplaceAll(line, " ", "")


It introduces another bug with `license = "Apache or MIT".

knqyf263 · 2024-04-23T09:28:49Z

pkg/fanal/analyzer/language/c/conan/conan.go

+				if license != "" {
+					break
+				}
+			} else if strings.HasPrefix(line, "license") { // cf. https://docs.conan.io/1/reference/conanfile/attributes.html#license


It should be more strict. Otherwise, it matches other than the license line. That's why I suggested regexp this time.
https://github.com/conan-io/conan-center-index/blob/55703762e2ca0560fd44d9008e00e61a0a64b23c/recipes/zlib/all/conanfile.py#L90

it looks like we can't avoid using regexp...

If possible, I don't want to use regexp. I'm still considering a way to parse the license line without regexp, but...

take a look this -https://go.dev/play/p/yWIioLCblQ9

Yes, it's better.

used strings.Cut - 9b5d8ce + 7bedafb

knqyf263 · 2024-04-23T09:29:36Z

pkg/fanal/analyzer/language/c/conan/conan.go

+		for scanner.Scan() {
+			line := strings.TrimSpace(scanner.Text())
+
+			if strings.HasPrefix(line, "name") { // cf. https://docs.conan.io/1/reference/conanfile/attributes.html#name


knqyf263 · 2024-04-23T09:37:35Z

pkg/fanal/analyzer/language/c/conan/conan.go

+
+func licensesFromCache() (map[string]string, error) {
+	required := func(filePath string, d fs.DirEntry) bool {
+		return filepath.Base(filePath) == "conanfile.py"


Isn't conanfile.txt used?

cache dir contains only conanfile.py files:

root@3fcddc7fe2e5:/app# find /root/.conan -name "conanfile.*" /root/.conan/data/zlib/1.3.1/_/_/export/conanfile.py

Is conanfile.py required? I'm wondering if we just don't find a project using conanfile.txt as most projects use conanfile.py. But we can handle conanfile.txt if we find such a case.

IIUC only conanfile.py contains attributes - https://docs.conan.io/2/reference/conanfile_txt.html#conanfile-txt

Therefore, we can't detect package name/license from conanfile.txt files.
Then we don't need to parse the conanfile.txt files.

knqyf263 · 2024-04-23T16:49:28Z

pkg/fanal/analyzer/language/c/conan/conan.go

+
+func licensesFromCache() (map[string]string, error) {
+	required := func(filePath string, d fs.DirEntry) bool {
+		return filepath.Base(filePath) == "conanfile.py"


Is conanfile.py required? I'm wondering if we just don't find a project using conanfile.txt as most projects use conanfile.py. But we can handle conanfile.txt if we find such a case.

knqyf263 · 2024-04-23T16:53:41Z

pkg/fanal/analyzer/language/c/conan/conan.go

+	// e.g. `license = "Apache or MIT"` -> ` "Apache or MIT"` -> `"Apache or MIT"` -> `Apache or MIT`
+	if name, v, ok := strings.Cut(line, "="); ok && strings.TrimSpace(name) == attributeName {
+		attr := strings.TrimSpace(v)
+		return strings.TrimPrefix(strings.TrimSuffix(attr, "\""), "\"")


strings.Trim seems better.

Suggested change

return strings.TrimPrefix(strings.TrimSuffix(attr, "\""), "\"")

return strings.Trim(attr, `"`)

Changed in 0d67966

DmitriyLewen added 3 commits March 15, 2024 14:20

feat(conan): detect licenses using cache dir

441eaa7

test(conan): update prev tests, add new test with cache dir

052f193

docs: update conan page

d54d1ce

DmitriyLewen self-assigned this Mar 15, 2024

DmitriyLewen changed the title ~~feat(conan): add license support~~ feat(c): add license support for conan lock files Mar 15, 2024

DmitriyLewen added 4 commits March 15, 2024 16:24

fix: check empty app

2b06478

fix test

eb3d53d

test: fix Test_conanLockAnalyzer_Required test

58efb64

fix: sort libs

4a56a0d

DmitriyLewen marked this pull request as ready for review March 19, 2024 10:42

DmitriyLewen requested a review from knqyf263 as a code owner March 19, 2024 10:42

DmitriyLewen added 2 commits April 15, 2024 10:25

Merge branch 'main' of github.com:DmitriyLewen/trivy into feat-conan/…

9cc7e88

…license

refactor after merge main branch

61af875

knqyf263 reviewed Apr 22, 2024

View reviewed changes

DmitriyLewen mentioned this pull request Apr 23, 2024

feat(vuln): Handle scanning conan v2.x lockfiles #6357

Merged

6 tasks

refactor: remove extra spaces

89003d6

knqyf263 reviewed Apr 23, 2024

View reviewed changes

DmitriyLewen added 3 commits April 23, 2024 16:45

refactor: use strings.Cut for attributes

9b5d8ce

refactor: remove bool output for detectAttribute

7bedafb

test: add tests for detectAttribute

18b1383

knqyf263 reviewed Apr 23, 2024

View reviewed changes

refactor: use strings.Trim

0d67966

knqyf263 approved these changes Apr 24, 2024

View reviewed changes

knqyf263 added this pull request to the merge queue Apr 24, 2024

Merged via the queue into aquasecurity:main with commit 5dd9bd4 Apr 24, 2024
17 checks passed

fl0pp5 pushed a commit to altlinux/trivy that referenced this pull request May 6, 2024

feat(c): add license support for conan lock files (aquasecurity#6329)

25ee3c8

aqua-bot mentioned this pull request May 29, 2024

release: v0.52.0 [main] #6809

Merged

DmitriyLewen mentioned this pull request Jun 17, 2024

fix(c): don't skip conan files from file-patterns and scan .conan2 cache dir #6949

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(c): add license support for conan lock files #6329

feat(c): add license support for conan lock files #6329

DmitriyLewen commented Mar 15, 2024

knqyf263 Apr 22, 2024 •

edited

Loading

knqyf263 Apr 22, 2024

DmitriyLewen Apr 23, 2024

DmitriyLewen Apr 23, 2024

knqyf263 Apr 23, 2024 •

edited

Loading

DmitriyLewen Apr 23, 2024

knqyf263 Apr 23, 2024

DmitriyLewen Apr 23, 2024

knqyf263 Apr 23, 2024

knqyf263 Apr 23, 2024

DmitriyLewen Apr 23, 2024

knqyf263 Apr 23, 2024

DmitriyLewen Apr 23, 2024 •

edited

Loading

knqyf263 Apr 23, 2024

DmitriyLewen Apr 23, 2024 •

edited

Loading

knqyf263 Apr 23, 2024

knqyf263 Apr 23, 2024 •

edited

Loading

DmitriyLewen Apr 23, 2024

knqyf263 Apr 23, 2024

DmitriyLewen Apr 24, 2024

knqyf263 Apr 23, 2024

knqyf263 Apr 23, 2024 •

edited

Loading

DmitriyLewen Apr 24, 2024

	return strings.TrimPrefix(strings.TrimSuffix(attr, "\""), "\"")
	return strings.Trim(attr, `"`)

feat(c): add license support for conan lock files #6329

feat(c): add license support for conan lock files #6329

Conversation

DmitriyLewen commented Mar 15, 2024

Description

Related issues

Checklist

knqyf263 Apr 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DmitriyLewen Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 Apr 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knqyf263 Apr 22, 2024 •

edited

Loading

knqyf263 Apr 23, 2024 •

edited

Loading

DmitriyLewen Apr 23, 2024 •

edited

Loading

DmitriyLewen Apr 23, 2024 •

edited

Loading

knqyf263 Apr 23, 2024 •

edited

Loading

knqyf263 Apr 23, 2024 •

edited

Loading