Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to open JAR files #3514

Closed
knqyf263 opened this issue Feb 1, 2023 · 5 comments · Fixed by #3582
Closed

Unable to open JAR files #3514

knqyf263 opened this issue Feb 1, 2023 · 5 comments · Fixed by #3582
Assignees
@DmitriyLewen
Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@knqyf263
Copy link
Collaborator

knqyf263 commented Feb 1, 2023

Description

Java scanning may lead to the following error.

failed to analyze file: failed to analyze usr/lib/jvm/java-1.8-openjdk/lib/tools.jar: unable to open usr/lib/jvm/java-1.8-openjdk/lib/tools.jar: failed to open: unable to read the file: stream error: stream ID 9; PROTOCOL_ERROR; received from peer

Currently, we're investigating this issue. As a temporary mitigation, you may be able to avoid this issue by downloading the Java DB in advance. --download-java-db-only is available in v0.37.0+.

$ trivy image --download-java-db-only
2023-02-01T16:57:04.322+0900    INFO    Downloading the Java DB...
$ trivy image [YOUR_JAVA_IMAGE]
@knqyf263 knqyf263 added kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Feb 1, 2023
@knqyf263 knqyf263 mentioned this issue Feb 1, 2023
Closed
@DmitriyLewen DmitriyLewen mentioned this issue Feb 1, 2023
Closed
6 tasks
sambles added a commit to OasisLMF/OasisPlatform that referenced this issue Feb 1, 2023
@sambles
fix for trivy issue - aquasecurity/trivy#3514
21afc42
sambles added a commit to OasisLMF/OasisPlatform that referenced this issue Feb 1, 2023
@sambles
Revert "fix for trivy issue - aquasecurity/trivy#3514"
88a8f2c 
This reverts commit 21afc42.
@nickbroon nickbroon mentioned this issue Feb 1, 2023
Closed
@StevDa86
Copy link

StevDa86 commented Feb 2, 2023 •
edited
Loading

Full log file: trivy_debug.log

Trivy version 0.36.1

Here is a debug log from a run which has the same error. Trivy just stops working.
Its wired cause some times the same image works and a few hours later not.

I also tested it with different images.

`2023-02-02T13:27:35.337Z DEBUG Parsing Java artifacts... {"file": "app/lib/netty-transport-native-kqueue-4.1.79.Final-osx-x86_64.jar"}
2023-02-02T13:27:35.694Z DEBUG Parsing Java artifacts... {"file": "app/lib/netty-transport-native-unix-common-4.1.79.Final.jar"}
2023-02-02T13:27:35.709Z DEBUG Parsing Java artifacts... {"file": "app/lib/nimbus-jose-jwt-9.22.jar"}
2023-02-02T13:27:35.724Z FATAL image scan error:
github.com/aquasecurity/trivy/pkg/commands/artifact.Run
/home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:397

  • scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
    /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:257
  • scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
    /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:588
  • failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
    /home/runner/work/trivy/trivy/pkg/scanner/scan.go:146
  • analyze error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.Inspect
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:127
  • failed to analyze layer: :
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect.func1
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:229
  • walk error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspectLayer
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:284
  • failed to process the file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.LayerTar.Walk
    /home/runner/work/trivy/trivy/pkg/fanal/walker/tar.go:90
  • failed to analyze file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.LayerTar.processFile
    /home/runner/work/trivy/trivy/pkg/fanal/walker/tar.go:104
  • failed to analyze app/lib/oauth2-oidc-sdk-9.35.jar:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspectLayer.func1
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:279
  • unable to open app/lib/oauth2-oidc-sdk-9.35.jar:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer.AnalyzerGroup.AnalyzeFile
    /home/runner/work/trivy/trivy/pkg/fanal/analyzer/analyzer.go:407
  • failed to open:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*cachedFile).Open
    /home/runner/work/trivy/trivy/pkg/fanal/walker/cached_file.go:65
  • unable to read the file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*cachedFile).Open.func1
    /home/runner/work/trivy/trivy/pkg/fanal/walker/cached_file.go:58
  • unexpected EOF`

sambles added a commit to OasisLMF/OasisPlatform that referenced this issue Feb 2, 2023
@sambles
Fixes for Github actions - Platform 2.x (#717)
87b22a0 
* Github actions CI/CD update (#711)

* Add working notes on docker image caching

* Add placeholder files

* test sarif report

* Fix fmt

* needs repo

* f

* scan all requ files

* report update

* fix report rename

* complete repo / python scanning

* complete repo / python scanning

* Fix URL append

* Fix URL append take2

* unit test and report with Junit

* apt install sudo

* Add report file

* Fix report path

* try another formatter

* Switch to python setup

* fix

* where is report

* f

* just tox

* switch to py3.10

* switch to py3.10

* Test build matrix

* Fix output?

* Create new file for piwind build

* remove piwind

* Ignore scan errors on debian image

* Fix image ref

* f

* Add schema builder

* typo fix

* Fix missing dir

* fix

* time image size

* time image size

* fix

* fix again

* Add version workflow

* Move and update helper scripts

* missed params in img check

* Test images for wasted space

* Fix bash?

* fix

* Add version fixes

* Draft script to clean-up workflow runs

* tmp

* outline for release workflow

* Test workflow connections

* Add missing triggers

* Fix booleans

* test workaround

* test workaround

* try 1/0 for  bool

* How are booleans this broken in GH actions

* Update schema test script loc

* fix missing "/"

* Fix mount location

* print needs context

* no easy way to output from matrix of jobs, missing ref, create image names externally

* try from env

* no dice, env not in context

* ffs ..

* Fix schema scan

* where is the build output image names?

* Try unconnected output image names

* ????!?

* Give up.. hardcode output vars

* Give up.. hardcode output vars

* fix docker push

* disable push and force retest

* Force retest

* try build and push for plat

* force retest

* re-test

* Sorry for all the spam Matt ~ force retest

* Add release checks and image re-tags

* Fix align

* Fix syntax

* Test Release script

* fix

* re-test

* re-test

* fix tags

* fix tags

* fix branchs

* fix prev-release find steps

* fix?

* Test find release script

* Try running bash from script

* weird pathing, where is the script?

* Fix dir overwrite

* try again

* Fix for ktools tags which start with "v"

* Add docker push

* Fix script arg

* forece retest

* forece retest

* retest

* again

* Fix schmea and version checks

* Fix test branch + content type

* why no match?

* fix ver checks

* Fix oasislmf version check

* Add overrides to auto-fetch the correct current/prev versions

* fix ifs

* fix

* Fix create Release notes

* Move changelog script to OasisPlatform

* Add missing requ file

* Fix changelog script call

* Plz just publish

* Fix bad path var

* Fix release path

* Tidy & force retest

* Add image summary to build

* Fix summary

* Ready publish script for merge

* dont push "latest" tags on backport branches

* Draft image testing

* Test piwind matrix testing

* fix

* image build - fix checkout on PR

* fix

* Add option to skip scan image on build

* Test s3 piwind run from Platform

* Fix?

* test - set correct piwind branch

* force retest

* Retest with wait for portfolio

* If scan skipped also skip image size testing

* try only s3

* test

* Try tests without matrix

* Update stored results/logs + add extra tests

* Update test-piwind.yml

* Try smaller test

* Skip S3 check

* Skip S3 check

* Test with updated wait fixture

* Fix worker check

* Is the hang from debug mode?

* update

* Run in parallel

* Tidy up

* Update name

* Add piwind branch select

* fix script branch

* Add missing branch output

* remove placeholder files

* Update badges

* tidy names

* Add PR / push triggers

* Add manual trigger to image tests

* Fix env context error

* fix syntax

* Update trigger branches and enable CVE error on hit

* Fix scan inputs

* Update github workflows with versions from c8542c9

* Run Autopep8

* Fix unused imports - Flake8

* Collect extra lookup files (#731)

* Keys file merge

Fix parquet

Fix file merging

* Update file merge

* Fix code-ql post merge

* Switch branch targets

* Fix unit testing

* Fix find prev release

* Test keycloak and worker-controller scan

* Fix step name

* fix for trivy issue - aquasecurity/trivy#3514

* Revert "fix for trivy issue - aquasecurity/trivy#3514"

This reverts commit 21afc42.

* Wait for trivy bug fix

* Fix test_analysis_model.py

* Fix missing import + pep

* Add worker controller to build

* Add publish for worker controller

* Test plat2 compose in piwind

* Fix test images

* Try keycloak scan again

* Add workflow for scanning external images

* fix

* Fix external img scanning

* Fix worker controller build

* Fix missing mapping

* Switch to fail-fast false

* More fixes

* Add retry for failed model reg tasks

* Revert "Add retry for failed model reg tasks"

This reverts commit ebecb93.

* Retry worker reg, -- dont dup config

* Add workaround to align platform-2.0 branches with piwind

* Fix merge errors

* Fix missing piwind branch output
@DmitriyLewen
Copy link
Contributor

Hello @StevDa86
Thanks!

We know about this bug and can reproduce "unexpected EOF".
We are currently working on it.

@3XC1T3D
Copy link

3XC1T3D commented Feb 7, 2023 •
edited
Loading

Full log file: trivy_debug.log

Trivy version 0.36.1

Here is a debug log from a run which has the same error. Trivy just stops working. Its wired cause some times the same image works and a few hours later not.

I also tested it with different images.

`2023-02-02T13:27:35.337Z DEBUG Parsing Java artifacts... {"file": "app/lib/netty-transport-native-kqueue-4.1.79.Final-osx-x86_64.jar"} 2023-02-02T13:27:35.694Z DEBUG Parsing Java artifacts... {"file": "app/lib/netty-transport-native-unix-common-4.1.79.Final.jar"} 2023-02-02T13:27:35.709Z DEBUG Parsing Java artifacts... {"file": "app/lib/nimbus-jose-jwt-9.22.jar"} 2023-02-02T13:27:35.724Z FATAL image scan error: github.com/aquasecurity/trivy/pkg/commands/artifact.Run /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:397

  • scan error:
    github.com/aquasecurity/trivy/pkg/commands/artifact.(*runner).scanArtifact
    /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:257
  • scan failed:
    github.com/aquasecurity/trivy/pkg/commands/artifact.scan
    /home/runner/work/trivy/trivy/pkg/commands/artifact/run.go:588
  • failed analysis:
    github.com/aquasecurity/trivy/pkg/scanner.Scanner.ScanArtifact
    /home/runner/work/trivy/trivy/pkg/scanner/scan.go:146
  • analyze error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.Inspect
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:127
  • failed to analyze layer: :
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspect.func1
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:229
  • walk error:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspectLayer
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:284
  • failed to process the file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.LayerTar.Walk
    /home/runner/work/trivy/trivy/pkg/fanal/walker/tar.go:90
  • failed to analyze file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.LayerTar.processFile
    /home/runner/work/trivy/trivy/pkg/fanal/walker/tar.go:104
  • failed to analyze app/lib/oauth2-oidc-sdk-9.35.jar:
    github.com/aquasecurity/trivy/pkg/fanal/artifact/image.Artifact.inspectLayer.func1
    /home/runner/work/trivy/trivy/pkg/fanal/artifact/image/image.go:279
  • unable to open app/lib/oauth2-oidc-sdk-9.35.jar:
    github.com/aquasecurity/trivy/pkg/fanal/analyzer.AnalyzerGroup.AnalyzeFile
    /home/runner/work/trivy/trivy/pkg/fanal/analyzer/analyzer.go:407
  • failed to open:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*cachedFile).Open
    /home/runner/work/trivy/trivy/pkg/fanal/walker/cached_file.go:65
  • unable to read the file:
    github.com/aquasecurity/trivy/pkg/fanal/walker.(*cachedFile).Open.func1
    /home/runner/work/trivy/trivy/pkg/fanal/walker/cached_file.go:58
  • unexpected EOF`

we have the same issue by scanning a Tomcat Image. Did it help to download the Java DB in advance?

EDIT: the workaround to download the Java DB in advance helped in our scenario

@Momotoculteur
Copy link

Hi guys,

i can reproduce & get the same behaviour

This was referenced Feb 9, 2023
Merged
Closed
@knqyf263 knqyf263 mentioned this issue Feb 22, 2023
Closed
sambles added a commit to OasisLMF/OasisPlatform that referenced this issue Feb 28, 2023
@sambles @carlfischerjba @awsbuild
Merge pull request #761 from OasisLMF/release/2.1.0
6dd90eb 
* Fix/CVE issues plat 2 (#648)

* Test trivy

* Updt

* Updt

* python CVE

* Updt

* Updt

* User mapped docker

* Update requ and pip oasislmf to latest LTS

* Update unittests to python3.10 + add user mapping

* Fix deps

* Update images to 22.04

* Update unittest base image

* Clean up older docker files

* Docker user map for schema tester

* Fix unittests

* Fix unittests

* Fix unittests

* fix unittests

* Run containers as non root

* Update pandas

* Fix build error installed cffi

* Non-root containers -- wip not building yet

* Lock down server image, no package installs with "server" user

* Fix pyarrow install error

* Fix worker build + lockdown

* Fix

* lockdown worker-controller

* Update docker-compose

* Fix $PATH

* Fix celery error

* Fix file access error

* fix

* Fix perms

* Fix Azure server crash

* Fix Azure server crash

* Fix root access for ONLY updating the hosts file -- azure crash issue

* Set store file w/o setting permistions

* Fix writes to root mounted fs by non-root accounts

* fix

* Update .dockerignore

* Fix twisted

* Fix wasted space

* Fix

* Fix unittest docker runner

* Fix arti storage

* test fix

* Try again

* Feature/main branch ports (#651)

* Feature/1018 default samples (#620)

* Add pkg-config for ktools build

* Load model_settings on run analysis - for default samples

* Update analysis schema

* Update model schema

* Feature/608 settings templates (#621)

* First pass

Swapped layout for nested resources

Fix serializers

* Fix routers and added description field

* Update requirments

* Rename filename to name

* Update deploy script

* Update tester

* Fix/list views order (#653)

* Option 1 - force order in VerifyGroupAccessModelViewSet

* Option 2 - Force the order via the model Meta field

* try 2

* Revert "Option 1 - force order in VerifyGroupAccessModelViewSet"

This reverts commit 883ba3c.

* remove comments

* Feature/filter options subtasks (#655)

* Add filterset for subtasks, and param fiters on analysis "sub_task_list"

* Fix template view

* Fix template view

* Fixed?

* Fix slug param and tidy

* Fix

* Fix/631 autoscale non zero (#656)

* Fix boolean arg loading

* Remove debug prints

* Fixes for work sub-tasks  (#657)

* Draft task context logger

* Update compose for debug

* Draft task log storage 2

* Clean up task log storage

* Create dir in workers for task logging "/var/log/oasis/tasks"

* Create timestamps before response queued

* Fix model_settings_json to abs path

* Add verbose logging

* Fix/keycloak auth proxy (#659)

* test X-Forwarded-For

* Fix parse error

* clean up

* Set proxy idle timeout to 2h

* Set keepalive options

* Final?

* Skip secret check in image scans

* Update gen scan

* Fix/loclines lessthan chunks (#660)

* Add check for loc lines

* Fix error msg

* Fix unittests

* Fix/646 websocket error handling (#662)

* Test workaround from GH issues
* Test worker-controller more updates

* Pin Oasislmf for output fix & fix logging

* Run update requirments as root

* Feature/pre analysis hook (#663)

* Basic pre-analysis hook

* tidy

* Fix loading

* Fix portfolio file removal

* Clean up tmp pre-analysis-hook exposure files on failed run

* Update oasislmf package pin

* Fix/output kat chunking (#665)

* Fix output kats and append logs to sub-task errors

* Add max chunks option

* Migrations

* Tidy up

* Pin new oasislmf fix

* Add max chunks to controller

* rm debug trace

* Fix for gulpy

* own conf.ini

* own var for jba

* Fix large file download/upload timeout  (#685)

* Scrap all changes and lay groundwork for two server containers

Switch socket port to 8001

Add uwsgi to requ

switch default command to wsgi

Revert "Add uwsgi to requ"

This reverts commit ae7f108.

Install uwsgi as binary package

Revert "Revert "Add uwsgi to requ""

This reverts commit 2f82ddf.

Add wsgi

Revert "Revert "Revert "Add uwsgi to requ"""

This reverts commit c43f2b3.

try gunicorn

fix routing

Increase timeout and match worker count to system cores

Increased timeout fixed download

test charts

Fix API azure access

Fixed DNS routing to websocket container

Fix token access issue?

Fix sub_path enable/disable

Disable "/api/ sub-path for internal web-socket server"

websocket server has no connection to keycloak for auth, try using the current oasisServer options

Fix?

fix

ugh

shot in the dark

Remove added client

use keycloak settings from server values

* Add option for ssl redis (#687)

* Add option for ssl redis

* Update configMaps

* syntax fix

* Add missing ssl data in configMap

* FIx ssl data in ConfigMap

* When running with REST api in Subpath, also work with root pathing

* Fix/auth timeout (#697)

* Test updated settings

* bump UI ver

* Revert "Test updated settings"

This reverts commit e5b6b3c.

* Increase timeout to 1d

* Hot fix for exposure file content type "application/vnd.ms-excel" (#700)

* Update/CVE versions (#708)

* Update packages with CVE issues

Update worker-controller

fix joblib and oauthlib

fix django

django-celery-results

pandas

ruamel.yaml

parso and distlib

SQLalc

Updated package virtualenv

Updated package filelock

Updated package azure-storage-blob

Updated package coverage

Updated package django-request-logging

Updated package drf-yasg

Updated package numpy

Updated package scipy

Updated package waitress

Updated package ods-tools

Updated package psycopg2-binary

Updated package scikit-learn

Add update requ by package script

Updated package oasislmf

Updated package celery==5.*

Updated package numpy

* Fix worker-controller crash with py3.10

* Update model settings schema

* Update to RC of oasislmf

* Revert "Update model settings schema"

This reverts commit b51e1a2.

* Revert "Update to RC of oasislmf"

This reverts commit 3eb9dad.

* Base case, update keycloak to most recent v15 - CVE (102) 32 High, 94 fixable

* Always include ssl in configmap

* Always include ssl in configmap (#719)

* Add support for registry credentials

* Remove redundant range

* Collect extra lookup files (#731)

* Keys file merge

Fix parquet

Fix file merging

* Update file merge

* Fixes for Github actions - Platform 2.x (#717)

* Github actions CI/CD update (#711)

* Add working notes on docker image caching

* Add placeholder files

* test sarif report

* Fix fmt

* needs repo

* f

* scan all requ files

* report update

* fix report rename

* complete repo / python scanning

* complete repo / python scanning

* Fix URL append

* Fix URL append take2

* unit test and report with Junit

* apt install sudo

* Add report file

* Fix report path

* try another formatter

* Switch to python setup

* fix

* where is report

* f

* just tox

* switch to py3.10

* switch to py3.10

* Test build matrix

* Fix output?

* Create new file for piwind build

* remove piwind

* Ignore scan errors on debian image

* Fix image ref

* f

* Add schema builder

* typo fix

* Fix missing dir

* fix

* time image size

* time image size

* fix

* fix again

* Add version workflow

* Move and update helper scripts

* missed params in img check

* Test images for wasted space

* Fix bash?

* fix

* Add version fixes

* Draft script to clean-up workflow runs

* tmp

* outline for release workflow

* Test workflow connections

* Add missing triggers

* Fix booleans

* test workaround

* test workaround

* try 1/0 for  bool

* How are booleans this broken in GH actions

* Update schema test script loc

* fix missing "/"

* Fix mount location

* print needs context

* no easy way to output from matrix of jobs, missing ref, create image names externally

* try from env

* no dice, env not in context

* ffs ..

* Fix schema scan

* where is the build output image names?

* Try unconnected output image names

* ????!?

* Give up.. hardcode output vars

* Give up.. hardcode output vars

* fix docker push

* disable push and force retest

* Force retest

* try build and push for plat

* force retest

* re-test

* Sorry for all the spam Matt ~ force retest

* Add release checks and image re-tags

* Fix align

* Fix syntax

* Test Release script

* fix

* re-test

* re-test

* fix tags

* fix tags

* fix branchs

* fix prev-release find steps

* fix?

* Test find release script

* Try running bash from script

* weird pathing, where is the script?

* Fix dir overwrite

* try again

* Fix for ktools tags which start with "v"

* Add docker push

* Fix script arg

* forece retest

* forece retest

* retest

* again

* Fix schmea and version checks

* Fix test branch + content type

* why no match?

* fix ver checks

* Fix oasislmf version check

* Add overrides to auto-fetch the correct current/prev versions

* fix ifs

* fix

* Fix create Release notes

* Move changelog script to OasisPlatform

* Add missing requ file

* Fix changelog script call

* Plz just publish

* Fix bad path var

* Fix release path

* Tidy & force retest

* Add image summary to build

* Fix summary

* Ready publish script for merge

* dont push "latest" tags on backport branches

* Draft image testing

* Test piwind matrix testing

* fix

* image build - fix checkout on PR

* fix

* Add option to skip scan image on build

* Test s3 piwind run from Platform

* Fix?

* test - set correct piwind branch

* force retest

* Retest with wait for portfolio

* If scan skipped also skip image size testing

* try only s3

* test

* Try tests without matrix

* Update stored results/logs + add extra tests

* Update test-piwind.yml

* Try smaller test

* Skip S3 check

* Skip S3 check

* Test with updated wait fixture

* Fix worker check

* Is the hang from debug mode?

* update

* Run in parallel

* Tidy up

* Update name

* Add piwind branch select

* fix script branch

* Add missing branch output

* remove placeholder files

* Update badges

* tidy names

* Add PR / push triggers

* Add manual trigger to image tests

* Fix env context error

* fix syntax

* Update trigger branches and enable CVE error on hit

* Fix scan inputs

* Update github workflows with versions from c8542c9

* Run Autopep8

* Fix unused imports - Flake8

* Collect extra lookup files (#731)

* Keys file merge

Fix parquet

Fix file merging

* Update file merge

* Fix code-ql post merge

* Switch branch targets

* Fix unit testing

* Fix find prev release

* Test keycloak and worker-controller scan

* Fix step name

* fix for trivy issue - aquasecurity/trivy#3514

* Revert "fix for trivy issue - aquasecurity/trivy#3514"

This reverts commit 21afc42.

* Wait for trivy bug fix

* Fix test_analysis_model.py

* Fix missing import + pep

* Add worker controller to build

* Add publish for worker controller

* Test plat2 compose in piwind

* Fix test images

* Try keycloak scan again

* Add workflow for scanning external images

* fix

* Fix external img scanning

* Fix worker controller build

* Fix missing mapping

* Switch to fail-fast false

* More fixes

* Add retry for failed model reg tasks

* Revert "Add retry for failed model reg tasks"

This reverts commit ebecb93.

* Retry worker reg, -- dont dup config

* Add workaround to align platform-2.0 branches with piwind

* Fix merge errors

* Fix missing piwind branch output

* OED fixes for platform2  (#744)

* fix loading location data in worker

* replace package updates

* Update server - python requ and ods-tools>3.0.1

* Added OED validation on file upload, and updated ods-tools package to 3.0.1 (#724)

* test file return format conv

This reverts commit 8c0de7e.

* Draft option to validate oed files on upload

* validate on upload - wip

* Note for later

* Draft serializer to return portolio validation status

* WIP GET validated files

* POST portfolio validate

* Add validation url param

* read validation option, url/settings.py

* Set ods-tools 3.0.1

* fix unittests

* Fix validation on parquet file uploads

* Fix handling of invalid data upload

* Fix typos and missing docs strings

* remove dup func

* Turn valadation on by default

* Add validation unit testing

* Add test_all_exposure__are_valid

* set set_portolio_valid as portfolio method instead of function

* Move VALIDATION_CONFIG to settings.py

* Update Swagger to show validate on upload default=true

* Add migrations + fix swagger schema

* Fix pep8 and flake issues

* Fix issues with merge

* flake8

* Fix location file loading for oasislmf==1.27

* Fix

* CVE scans are raising issues but Actions marked as passed (#746)

* Is it fail fast?

* Revert "Is it fail fast?"

This reverts commit b7fceae.

* is output report masking error?

* Run Trivy twice, once for error status, then after to log all issues

* Log all unfixed issues

* Fix

* Same for platform scanning

* Test cache dir

* cache dosnt speed up 2nd run

* Only fail crit or high

* Keep a copy of requirments in final image (needed for scanning)

* No need to run seveal repo scans, issues should be picked up at image build

* Update/oasislmf 1.27 (#745)

* tidy up workflows

* Set worker to use oasislmf 1.27

* Set debug=1

* Fix eltcalc output

* Run all piwind tests

* Fix loading default - PORTFOLIO_UPLOAD_VALIDATION

* Fix test_portfolio.py

* Update kube-prometheus-stack (#748)

* Try switching kube-prometheus-stack

* Try 44.3.1

* Update helm dependency build + fix values

* Add draft workflow of helm

* update

* Update notes

* Set ods-tools 3.0.2 (#753)

* Set ods-tools 3.0.2

* Dont use ods-tools for format convention

If OED exposure is passed though ods-tools extra col will be added which returns different data based on the base format

Example:
  POST: CSV file -> portfolios/{n}/location_file

  GET: portfolios/{n}/location_file?format=csv      (no blank required col inserted)
  GET: portfolios/{n}/location_file?format=parquet  (any missing TIV cols will be added before data is return)

Fix:
  Only add extra columns if file is validated

* Remove unsed import

* RiskLevel allowed blank, remove error from test

* fix unused import

* Update/keycloak (#709)

* test latest

* Set keycloak to 19.0.3-legacy

* Add missing placeholder

* Try 20.0.3

* Fix keycloak config for v20

test

Fix auth path and admin user

clear old files

clean templates/keycloak.yaml

* Fix openid - only works for "./manage runserver"

tmp

Try fresh export

Try adding scope into requests

Revert "Try adding scope into requests"

This reverts commit 0c4abb54530e41095e4c15eaabcc42ffcab550e2.

Revert "Revert "Try adding scope into requests""

This reverts commit 0ca52c7a1eea1aef37e5ed3e7ce3726e402858ba.

* Try fix for remapping keycloak host external -> internal

* Fix dockerfile

* pep8 fix

* Delete backup oasis-realm

* Update external image versions  (#757)

* Show misconfig but dont fail CI

* scan every 6h

* Fix cryptography

* Only create image CVE if exitcode is 1

* Try latest external images

* test hack to set PV permisstions

* Update name

* Delete, recreate oasis-server client, and update realm export

* Limit external image scanning to vuln

* Fix flanky token request

Token requests work with a fresh access token but fail on refresh due to internal -> external mapping
missing from 'src/server/oasisapi/auth/serializers.py'

```
  File "/home/server/.local/lib/python3.10/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/home/server/.local/lib/python3.10/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/var/www/oasis/src/server/oasisapi/auth/views.py", line 44, in post
    return super().post(request, *args, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework_simplejwt/views.py", line 43, in post
    serializer.is_valid(raise_exception=True)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/serializers.py", line 227, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/home/server/.local/lib/python3.10/site-packages/rest_framework/serializers.py", line 429, in run_validation
    value = self.validate(value)
  File "/var/www/oasis/src/server/oasisapi/auth/serializers.py", line 73, in validate
    response = requests.post(
  File "/home/server/.local/lib/python3.10/site-packages/requests/api.py", line 115, in post
    return request("post", url, data=data, json=json, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
  File "/home/server/.local/lib/python3.10/site-packages/requests/adapters.py", line 565, in send
    raise ConnectionError(e, request=request)

Exception Type: ConnectionError at /access_token/
Exception Value: HTTPSConnectionPool(host='ui.oasis.local', port=443): Max retries exceeded with url: /auth/realms/oasis/protocol/openid-connect/token (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fa9bb239db0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
```

* Fix missing severity

* Skip mysql, since postgres is default

* PEP

* Update python packages

* Fix

* Fixes for merging of distributed input files (#758)

* Fixes for merging of distributed input files

* Simplify condition

* Lint

* Fix settings file file loading for custom lookup (#670)

* remove "lookup_complex_config_json" param if no settings file given

* ++ version num

* Fix pep

* Align with branch backports/1.27.x (#762)

* Fix schema build workflow (#764)

* Fix schmea build

* Fix missing dir

* Set version 2.1.0

* Dont use latest tag for v2 images, avoid problems with current platform

* Update changelog

---------

Co-authored-by: Carl Fischer <[email protected]>
Co-authored-by: awsbuild <[email protected]>
@namandf
Copy link

namandf commented Mar 13, 2023

Hi,

Still seeing similar issues in latest version of Trivy #3807

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DmitriyLewen DmitriyLewen

Labels
kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
6 participants
@3XC1T3D @knqyf263 @Momotoculteur @StevDa86 @DmitriyLewen @namandf