Upgrade dependencies 2023-11-27

[github-actions]

• Update PyCharm image
  • Bump base image tag (only same Debian release), if possible
  • Bump upstream version, if possible
  • Bump internal version
  • Build and test new image locally with Azul's make format
  • Remove unused dependencies with high or critical CVEs
  • Push commit to GitHub (directly to master branch, no PR needed)
  • GH Action workflow succeeded
  • Image is available on DockerHub
• Update Elasticsearch image
  • Add a README.md
  • Bump base image tag (only minor and patch versions), if possible
  • Bump internal version
  • Build and test new image locally with Azul's make test
  • Remove unused dependencies with high or critical CVEs
  • Push commit to GitHub (directly to main branch, no PR needed)
  • GH Action workflow succeeded
  • Image is available on DockerHub
• Create Azul PR, connected to this issue, with …
  • … revert 8f0fcf4
  • … updates to issue template for this type of issue
  • … script for vulnerability report (sorted desc by vulnerability name, abbreviate severity as one letter)
  • … changes to requirements*.txt from open Dependabot PRs, one commit per PR
  • … update to Python (only patch versions)
  • … Updates to Terraform (only patch versions)
  • … new PyCharm image
  • … new Elasticsearch image
  • … update to Docker images (only minor and patch versions)
  • … update to GitLab & GitLab runner images
  • … update to ClamAV image
  • … update to GitLab AMI
• Delete obsolete image tags from DockerHub (but consider that prod may not use the latest image) …
  • … for PyCharm image @hannes-ucsc did that.
  • … for Elasticsearch image @hannes-ucsc did that.
• Created tickets for any deferred updates to …
  • … to next major or minor Python version
    • Update to Python 3.12.x #5736
  • … to next major Docker version
    • Docker 25 is not yet out of beta
  • … to next major or minor Terraform version
    • Update Terraform to 1.6.x #5687
• Post vulnerability report for anvilprod on this issue
  • To be done after PR is deployed to anvilprod

[dsotirho-ucsc]

Anvilprod Vulnerability report for 2023-12-06 added as a new tab to the Anvilprod Inspector Findings spreadsheet.

(.venv) daniel@Crispin ~/repo/azul3 $ python scripts/inspector-findings.py
2023-12-06 16:56:18,600    INFO MainThread __main__: Fetching all findings from AWS Inspector
2023-12-06 16:56:18,614    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2023-12-06 16:56:22,480    INFO MainThread __main__: Fetched 672 findings from AWS Inspector with any severity
2023-12-06 16:56:22,481    INFO MainThread __main__: Found 195 unique vulnerabilities with severity matching ['CRITICAL', 'HIGH']
2023-12-06 16:56:22,482    INFO MainThread __main__: Writing file: inspector-findings_2023-12-06.csv
2023-12-06 16:56:22,483    INFO MainThread __main__: Done.

[dsotirho-ucsc]

anvilprod vulnerability report for 2023-12-11 added as a new tab to the Anvilprod Inspector Findings spreadsheet.
The previous 2023-12-06 included inactive findings and has been removed from the spreadsheet.

$ python scripts/inspector-findings.py --severity CRITICAL HIGH
2023-12-11 15:31:51,889    INFO MainThread __main__: Fetching findings from AWS Inspector
2023-12-11 15:31:51,900    INFO MainThread botocore.credentials: Found credentials in shared credentials file: ~/.aws/credentials
2023-12-11 15:31:53,781    INFO MainThread __main__: Fetched 209 findings from AWS Inspector with severity ['CRITICAL', 'HIGH']
2023-12-11 15:31:53,782    INFO MainThread __main__: Found 170 unique vulnerabilities
2023-12-11 15:31:53,782    INFO MainThread __main__: Writing file: inspector-findings_2023-12-11.csv
2023-12-11 15:31:53,783    INFO MainThread __main__: Done.

[hannes-ucsc]

[hannes-ucsc]

Above is a screenshot of the most recent report with findings removed for images we don't use inside the boundary.

It's OK to post image vulnerability findings for Docker images. Which images we use is publicly available information (source code). Their vulnerabilities are also publicly listed on Dockerhub.

[hannes-ucsc]

The next report should sort the rows by severity and number of affected images. Triaging to see what this would entail.

[hannes-ucsc]

Lastly, it is a bit silly to just ignore checklist items that one doesn't have permissions for. Obviously, something needs to be done. The assignee of the ticket is responsible for that.