Update samples commit, refresh test data, fix refresh-test-data on macOS #508
Conversation
| mv $(OUT_DIR) $(basename $(OUT_DIR)) | ||
|
|
||
| prepare-samples: out/samples-$(SAMPLES_COMMIT) | ||
| cp -a test_data/. $(basename $(OUT_DIR)) | ||
|
|
||
| .PHONY: test | ||
| test: prepare-samples | ||
| go test $(shell go list ./... | grep -Ev "samples|test_data") |
There was a problem hiding this comment.
This was excluding the out/samples-... directories. Oddly enough, the test_data directory was still being run when I updated this to `grep -Ev "^samples$|test_data" so I updated this to the valid two directories with tests.
| @@ -108,21 +108,21 @@ addq ${malcontent} --format=simple \ | |||
| for f in $(find "${test_data}" -name "*.simple"); do | |||
| prog=$(echo $f | sed -e s#"${test_data}/"## -e s#\.simple\$##) | |||
| if [[ -f "${prog}" ]]; then | |||
| addq ${malcontent} --format=simple -o "${f}" analyze "${prog}" | |||
| addq ${malcontent} --format=simple --ignore-tags=harmless -o "${f}" analyze "${prog}" | |||
There was a problem hiding this comment.
I added these flags to match the test mc configuration.
Signed-off-by: egibs <[email protected]>
test_data/macOS/clean/ls.json
Outdated
| @@ -17,6 +17,18 @@ | |||
| "rpath" | |||
| ], | |||
| "Behaviors": [ | |||
| { | |||
There was a problem hiding this comment.
There seems to be some ordering oscillation going on here and it changes from run to run.
…fix test data Signed-off-by: egibs <[email protected]>
| @@ -27,6 +27,8 @@ func (r JSON) File(_ context.Context, _ *malcontent.FileReport) error { | |||
| } | |||
|
|
|||
| func (r JSON) Full(_ context.Context, rep *malcontent.Report) error { | |||
| // Drop the applied filters | |||
There was a problem hiding this comment.
We don't display this for any other renderer (though it's still set in the report) and it was causing weird diffs when testing JSON output.
| @@ -552,6 +553,11 @@ func Generate(ctx context.Context, path string, mrs yara.MatchRules, c malconten | |||
| fr.RiskScore = overallRiskScore | |||
| fr.RiskLevel = RiskLevels[fr.RiskScore] | |||
|
|
|||
| // Ensure that the behaviors are consistently sorted by ID | |||
There was a problem hiding this comment.
We rebuild the Behaviors slice from a map when working on overrides, so that's why we were seeing ordering issues. I can probably do this in handleOverrides but this ensures that the sorting happens right before we return the report.
egibs
force-pushed
the
update-samples
branch
2 times, most recently
from
470f8f6 to
fd5cf04
Compare
Signed-off-by: egibs <[email protected]>
|
Opened chainguard-dev/malcontent-samples#11 to fix the remaining tests (incorrect directory name is causing issues). |
Signed-off-by: egibs <[email protected]>
| if [ "$(shell uname)" = "Darwin" ]; then \ | ||
| tar xJvf "$$temp_path" -C $$(dirname "$$temp_path"); \ | ||
| elif [ "$(shell uname)" = "Linux" ]; then \ | ||
| tar xvf "$$temp_path" -C $$(dirname "$$temp_path"); \ |
There was a problem hiding this comment.
Linux should support tar xJvf too?
There was a problem hiding this comment.
I was getting tar: short read errors when using it for some reason (at least in Wolfi).
Signed-off-by: egibs <[email protected]>
Signed-off-by: egibs <[email protected]>
Closes: #503, #505
This PR updates the samples commit to the latest ref and fixes
.xzarchive extraction where some archives contain archives. If this is the case, we'll decompress the.xzfile and then extract the resulting archive viatar xJvf.I also fixed the test data refresh on macOS by re-adding
-n1to the final xargs command.It's likely that we'll see some JSON test failures so those will need to be resolved before this can merge.