Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ClamAV fails to complete scan #6022

Closed
dsotirho-ucsc opened this issue Mar 8, 2024 · 9 comments
Closed

ClamAV fails to complete scan #6022

dsotirho-ucsc opened this issue Mar 8, 2024 · 9 comments
Assignees
Labels
+ [priority] High compliance [subject] Information and software security debt [type] A defect incurring continued engineering cost demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team infra [subject] Project infrastructure like CI/CD, build and deployment scripts orange [process] Done by the Azul team

Comments

@dsotirho-ucsc
Copy link
Contributor

dsotirho-ucsc commented Mar 8, 2024

Since the upgrade of ClamAV from 1.2.1-27 to 1.3.0-37 on Feb 27th (for the upgrade dependencies ticket #5926, PR #5969), Clamscan have been failing on all deployments with the update.

Note: The ClamAV schedule was updated as part of a separate issue (PR #5943) on Feb 29th, however as seen in the logs below, the failures started after the version update and before the schedule update was applied.

CloudWatch logs on dev:

Screenshot 2024-03-08 at 5 43 10 PM

CloudWatch logs for a failing run on dev:

Feb 28 17:58:08 ip-172-71-0-215 docker: clamscan: ClamAV update process started at Wed Feb 28 17:58:08 2024
Feb 28 17:58:08 ip-172-71-0-215 docker: clamscan: daily database available for update (local version: 27195, remote version: 27199)
Feb 28 17:58:11 ip-172-71-0-215 docker: clamscan: Testing database: '/var/lib/clamav/tmp.86372efe5b/clamav-f02af4d30b13f416548f636de1feed88.tmp-daily.cld' ...
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: Database test passed.
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: daily.cld updated (version: 27199, sigs: 2054066, f-level: 90, builder: raynman)
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: bytecode database available for update (local version: 334, remote version: 335)
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: Testing database: '/var/lib/clamav/tmp.86372efe5b/clamav-ddb5190ef1bc551a22ec5b94c0d8c61c.tmp-bytecode.cld' ...
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: Database test passed.
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: bytecode.cld updated (version: 335, sigs: 86, f-level: 90, builder: raynman)
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: WARNING: Clamd was NOT notified: Can't connect to clamd through /tmp/clamd.sock: No such file or directory
Feb 28 17:58:23 ip-172-71-0-215 docker: clamscan: freshclam succeeded
Feb 28 18:27:21 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:27:55 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:28:03 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:28:09 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:28:16 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:28:22 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:11 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:21 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:27 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:32 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:39 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 18:29:45 ip-172-71-0-215 docker: clamscan: LibClamAV Warning: cli_scanxz: decompress file size exceeds limits - only scanning 105906176 bytes
Feb 28 20:15:13 ip-172-71-0-215 docker: clamscan: LibClamAV Error: cli_html_normalise: style chunk size underflow
Feb 28 21:39:18 ip-172-71-0-215 docker: clamscan: LibClamAV Error: cli_html_normalise: style chunk size underflow
Feb 28 22:08:40 ip-172-71-0-215 docker: clamscan: LibClamAV Error: cli_html_normalise: style chunk size underflow
Feb 28 23:38:14 ip-172-71-0-215 kernel: clamscan[28473]: segfault at 7fe7e1319372 ip 00007fe8345a4fde sp 00007ffdf4a03df0 error 4 in libclamav.so.12.0.2[7fe83441c000+5c1000]
Feb 28 23:38:15 ip-172-71-0-215 docker: clamscan: clamscan failed
Feb 28 23:38:15 ip-172-71-0-215 docker: clamscan: Segmentation fault (core dumped)
Feb 28 23:38:24 ip-172-71-0-215 systemd: clamscan.service: main process exited, code=exited, status=1/FAILURE
Feb 28 23:38:24 ip-172-71-0-215 systemd: Unit clamscan.service entered failed state.
Feb 28 23:38:24 ip-172-71-0-215 systemd: clamscan.service failed.

For reproduction, see comment.

@dsotirho-ucsc dsotirho-ucsc added the orange [process] Done by the Azul team label Mar 8, 2024
@dsotirho-ucsc dsotirho-ucsc self-assigned this Mar 8, 2024
@dsotirho-ucsc
Copy link
Contributor Author

Assignee to provide description

@dsotirho-ucsc
Copy link
Contributor Author

dsotirho-ucsc commented Mar 9, 2024

Confirmed the failure is due to ClamAV upgrade. ClamAV successfully completed on dev with version 1.2.1.27 temporarily deployed to the instance on Fri Mar 8th, and continued to succeed over the weekend with this older version.

Note: The failures from Feb 28th to Mar 8th occurred despite ClamAV being upgraded multiple times over this time frame: v1.3.0-37 (PR #5969) (merged Feb 27th), v1.3.0-40 (PR #5996) (merged Mar 4th), and v1.3.0-41 (PR #6019) (merged Mar 7th).

CloudWatch logs on dev:

Screenshot 2024-03-11 at 9 55 27 AM

@dsotirho-ucsc dsotirho-ucsc removed their assignment Mar 9, 2024
@dsotirho-ucsc
Copy link
Contributor Author

@hannes-ucsc: "Assignee to file partial PR that downgrades ClamAV. Ticket will then go to parked column until upstream fixes the segmentation fault."

@dsotirho-ucsc dsotirho-ucsc added bug debt [type] A defect incurring continued engineering cost infra [subject] Project infrastructure like CI/CD, build and deployment scripts compliance [subject] Information and software security + [priority] High labels Mar 11, 2024
@dsotirho-ucsc dsotirho-ucsc self-assigned this Mar 11, 2024
@dsotirho-ucsc
Copy link
Contributor Author

Clamscan is consistently crashing on the file

/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
[ec2-user@ip-172-21-0-99 ~]$ sudo grep -B 5 -A 2 "segfault at" /var/log/messages
--
Mar 19 02:12:22 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_CN/LC_MESSAGES/hg.mo
Mar 19 02:12:22 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_TW/LC_MESSAGES/hg.mo
Mar 19 02:12:22 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/it/LC_MESSAGES/hg.mo
Mar 19 02:12:22 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/fr/LC_MESSAGES/hg.mo
Mar 19 02:12:22 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
Mar 19 02:12:22 ip-172-21-0-99 kernel: clamscan[5635]: segfault at 7f09f4a4e2e2 ip 00007f0a47f90fde sp 00007ffdec354da0 error 4 in libclamav.so.12.0.2[7f0a47e08000+5c1000]
Mar 19 02:12:24 ip-172-21-0-99 docker: clamscan: Segmentation fault (core dumped)
Mar 19 02:12:24 ip-172-21-0-99 docker: clamscan: clamscan failed
--
Mar 19 07:26:23 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_CN/LC_MESSAGES/hg.mo
Mar 19 07:26:23 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_TW/LC_MESSAGES/hg.mo
Mar 19 07:26:23 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/it/LC_MESSAGES/hg.mo
Mar 19 07:26:23 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/fr/LC_MESSAGES/hg.mo
Mar 19 07:26:23 ip-172-21-0-99 docker: clamscan: Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
Mar 19 07:26:24 ip-172-21-0-99 kernel: clamscan[3479]: segfault at 7fd07d2592e2 ip 00007fd0d07c1fde sp 00007ffc532b2200 error 4 in libclamav.so.12.0.2[7fd0d0639000+5c1000]
Mar 19 07:26:25 ip-172-21-0-99 docker: clamscan: Segmentation fault (core dumped)
Mar 19 07:26:25 ip-172-21-0-99 docker: clamscan: clamscan failed

Reproduction:

[ec2-user@ip-172-21-0-99 ~]$ sudo /usr/bin/docker run --name clamscan --rm --volume /var/run/docker.sock:/var/run/docker.sock --volume /:/scan:ro --volume /mnt/gitlab/clamav:/var/lib/clamav:rw 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/clamav/clamav:1.3.0-41 /bin/sh -c "freshclam && echo freshclam succeeded || (echo freshclam failed; false) && clamscan --verbose --recursive --infected --allmatch=yes /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial && echo clamscan succeeded || (echo clamscan failed; false)"
ClamAV update process started at Tue Mar 19 07:42:33 2024
daily.cld database is up-to-date (version: 27218, sigs: 2055638, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cld database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
freshclam succeeded
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/grep.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/urllibcompat.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/formatter.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/scmposix.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/mergestate.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/metadata.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/exthelper.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/templatefilters.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/vfs.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/el/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/da/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/sv/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ru/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ro/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/de/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_CN/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_TW/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/it/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/fr/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
Segmentation fault (core dumped)
clamscan failed
[ec2-user@ip-172-21-0-99 ~]$ sudo /usr/bin/docker run --name clamscan --rm --volume /var/run/docker.sock:/var/run/docker.sock --volume /:/scan:ro --volume /mnt/gitlab/clamav:/var/lib/clamav:rw 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/clamav/clamav:1.3.0-41 /bin/sh -c "freshclam && echo freshclam succeeded || (echo freshclam failed; false) && clamscan --verbose --recursive --infected --allmatch=yes /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo && echo clamscan succeeded || (echo clamscan failed; false)"
ClamAV update process started at Tue Mar 19 07:51:28 2024
daily.cld database is up-to-date (version: 27218, sigs: 2055638, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cld database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
freshclam succeeded
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
Segmentation fault (core dumped)
clamscan failed

@hannes-ucsc
Copy link
Member

For demo, attempt to reproduce. Don't close issue (PR was partial). Instead park issue and file ticket with upstream, providing the file and a simplified reproduction, probably using a docker container. I tried reproducing on Apple Silicone with a homebrewed installation of ClamAV and was not able to.

@hannes-ucsc hannes-ucsc added the demo [process] To be demonstrated at the end of the sprint label Mar 21, 2024
@dsotirho-ucsc dsotirho-ucsc added the demoed [process] Successfully demonstrated to team label Apr 30, 2024
@dsotirho-ucsc
Copy link
Contributor Author

Don't close issue (PR was partial). Instead park issue and file ticket with upstream, providing the file and a simplified reproduction, probably using a docker container.

https://www.github.com/Cisco-Talos/clamav/issues/1262

@dsotirho-ucsc
Copy link
Contributor Author

Issue with ClamAV crashing is reported as fixed as of v1.3.1 https://www.github.com/Cisco-Talos/clamav/issues/1262 and was confirmed locally with the problematic file.

daniel@Crispin ~ $ docker run --name clamscan --rm --volume $PWD:/scan docker.io/clamav/clamav:1.3.1-49 /bin/sh -c "freshclam && clamscan --verbose --recursive --infected --allmatch=yes /scan/hg.mo"
Unable to find image 'clamav/clamav:1.3.1-49' locally
1.3.1-49: Pulling from clamav/clamav
4abcf2066143: Already exists
806eb0d476e2: Pull complete
813afc8c1511: Pull complete
7874a5ffbb63: Pull complete
4de4d31ec1e0: Pull complete
16f5f43f2b7a: Pull complete
034f2aa4e3ea: Pull complete
Digest: sha256:0f66a329bbaec3cc345a474568ae81d261af5ee01f5747c5acd1b489ff2a46b1
Status: Downloaded newer image for clamav/clamav:1.3.1-49
ClamAV update process started at Thu May  2 21:32:40 2024
daily database available for update (local version: 27252, remote version: 27263)
Testing database: '/var/lib/clamav/tmp.8ba2b10243/clamav-538f653a02c854fda957edf2cb87cc44.tmp-daily.cld' ...
Database test passed.
daily.cld updated (version: 27263, sigs: 2060257, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cvd database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
WARNING: Clamd was NOT notified: Can't connect to clamd through /tmp/clamd.sock: No such file or directory
Scanning /scan/hg.mo

----------- SCAN SUMMARY -----------
Known viruses: 8692087
Engine version: 1.3.1
Scanned directories: 0
Scanned files: 1
Infected files: 0
Data scanned: 4.13 MB
Data read: 1.14 MB (ratio 3.61:1)
Time: 24.387 sec (0 m 24 s)
Start Date: 2024:05:02 21:32:55
End Date:   2024:05:02 21:33:19

@dsotirho-ucsc
Copy link
Contributor Author

Waiting for #6212 (PR #6229) to be merged, and for the updated ClamAV to complete a successful scan.

@dsotirho-ucsc
Copy link
Contributor Author

ClamAV 1.3.1 is now on prod and scans are completing successfully.

Screenshot 2024-05-20 at 9 43 48 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
+ [priority] High compliance [subject] Information and software security debt [type] A defect incurring continued engineering cost demo [process] To be demonstrated at the end of the sprint demoed [process] Successfully demonstrated to team infra [subject] Project infrastructure like CI/CD, build and deployment scripts orange [process] Done by the Azul team
Projects
None yet
Development

No branches or pull requests

3 participants