Discussion: Antivirus on File Upload

[jeurgen]

ASVS recommends applications to scan files being obtained from untrusted sources for viruses:  
 
"12.4.2 Verify that files obtained from untrusted sources are scanned by antivirus 
scanners to prevent upload of known malicious content."
 
I am not entirely happy with this requirement and will be discussing advantages and disadvantages in this issue in order to launch a discussion on possible improvements. 
 
Usually uploaded files are not executed on the server-side. They therefore do not put server systems at risk. Risks evolve for application users when malicious files are downloaded and executed on the client-side. One could therefore suggest that files should be scanned by antivirus on the user endpoint only.  
 
However, I still see the following advantages of server-side AV scanning: 
 

• Users expect applications to take care of their security and implicitly assume this includes anti-virus scanning.  

• A malicious file may benefit from the reputation of a trusted platform. The probability of this file being executed by a victim might increase. 

 
On the other hand, there are also a number of disadvantages: 
 

• Increased attack surface: it is well-known that AV products are themselves often prone to vulnerabilities. These vulnerabilities might then be exploited through the application. 
   

• AV scanning can cause unexpected behaviour if, .e.g, uploaded files are stored in database blobs and AV then quarantines the entire database.  Of course, this would imply an AV misconfiguration or misuse, but I heard it happens in practice. 
   

• Files could be lost without the sender or receiver noticing. This of course depends on the particular implementation, but in practice the AV scan is not performed synchronously within the application itself but is performed by a third-party component.  

• Server-side AV scanning increases the overall complexity, as applications often do not offer out of the box interfaces for AV integration. A common approach is therefore to offload AV scanning to reverse proxies. While this is perfectly possible it still adds another layer of complexity with – depending on the load – multiple additional server systems being deployed. 

In conclusion, I believe that the disadvantages outweigh the advantages. I would therefore suggest revising the requirement for server-side AV scanning. A possible trade-off could be to warn users about infected files when downloading. This would leave most of the disadvantages of server-side scanning in place, but has two benefits: 
 
1. Files don’t get deleted unnoticed, if AV scanning was not synchronized during upload. 

2. AV scanning takes place at the time of downloading with up-to-date signatures. 

 
Google and Microsoft are following this approach with their cloud storage, for example. Dropbox and Xing don’t seem to do any antivirus scanning. To me both of these methods (‘warn on download’ and ‘do nothing’) seem preferable to AV-Scanning on file upload.  

[elarlang]

Increased attack surface: it is well-known that AV products are themselves often prone to vulnerabilities. These vulnerabilities might then be exploited through the application.

It's not different from using whatever other component or library.

AV scanning can cause unexpected behavior if, .e.g, uploaded files are stored in database blobs and AV then quarantines the entire database. Of course, this would imply an AV misconfiguration or misuse, but I heard it happens in practice.

AND

Files could be lost without the sender or receiver noticing. This of course depends on the particular implementation, but in practice the AV scan is not performed synchronously within the application itself but is performed by a third-party component.

If to use it as separate service on file upload AND download, it does not affect (and scan) application itself.

Something to take in account:

• user-uploaded files are stored in public folder (or content is accessible with HTTP request for non-authenticated user) - application is spreading known malware and will be blocked by browsers, ignored by Google etc. In this case, warning on download is not good. Additionally for potentially infected users, clear reputation damage.
• opening user-uploaded files is part of officer job, if to not filter them in advance, it increases possibility for targeted attacks against organization or company

In conclusion, I believe that the disadvantages outweigh the advantages.

I don't agree.

1. Files don't get deleted unnoticed, if AV scanning was not synchronized during upload. 

2. AV scanning takes place at the time of downloading with up-to-date signatures.

Even I don't agree with all the argumentation provided, I can see that those 2 requirements are something what current requirement needs.

What AV related requirements should cover:

• if user uploads malicious content - it's potential attack against your company/organization, you need to investigate it, so you can not just delete it, you need to store it but do not provide to site users
• public and "private" files may need different behavior
  • there is need to actively scan files in public automatically (to avoid serving known malicious content from trusted domain)
  • private files, served by program code for authenticated users, it's ok to do it on file download

If to put "scan again on file download" to this requirement, there may need to rethink about the subcategory also.

[danielcuthbert]

I appreciate the comments here, this is a topic that we've had amongst ourselves a few times and I do agree with the numerous points being made here, except the "disadvantages outweigh the advantages" one.

It's clear that 12.4.2 needs either a re-write to be clearer about what it is we are trying to achieve, or expanding it into a series of requirements dealing with the complexities of objects being uploaded and subsequent processing and use of said objects.

I'm leaning more towards the latter over the former, but keen to hear others views

[vanderaj]

If we can do a re-write without changing the meaning of 12.4.2, I'm happy to include this in 4.0.2. If it ends up being a huge change or multiple new items in 12.4, then this will have to wait for 4.1.

[elarlang]

Quickfix proposal (without creating new requirement) for v4.0.2

Current:

V12.4.2 Verify that files obtained from untrusted sources are scanned by antivirus scanners to prevent upload of known malicious content.

Proposal:

V12.4.2 Verify that files obtained from untrusted sources are scanned by antivirus scanners to prevent upload and hosting of known malicious content.

or "serving" instead of hosting?

[danielcuthbert]

Hosting is a bit oldfashioned right? yeah serving is more in line with what you would do if you had a delivery site for malicious code

[jeurgen]

Thanks for picking up on the issue. I think the discussion has turned into the wrong direction at the moment. I doubt the added value by appending "hosting" or "serving". The proposed changes do not deal with my actual concerns.

If AV scanning should be done mandatorily, then we should discuss about the right timing.

I still think it would give users a false sense of security, if they rely on AV Scans of a web application at the time of file upload. During the past months I dealt with several cases of emotet-like malware. Most of the time AVs were only able to detect the related files hours/days after the campaigns began.

As an attacker I would be happy to find a platform that does AV scanning on upload but not on download. It would give my malicious file the reputation of "AV Check passed" without having passed it at the relevant moment. The critical moment is when files from untrusted sources get delivered to other application users. I would agree on AV-scanning being done optionally also on file upload. But more important is the time of file delivery.

In the current proposal users are left unclear about the term "serving". This term would first need to be defined more precisely. Only adding the word "serving" would make the wording even more vague.

[elarlang]

1. please explain, how given proposal is wrong direction
2. what is your proposal?

Context - take in account, that for ASVS v4.0.2 the meaning for requirement text in big picture should stay the same. If we talk about separate requirements, then it's for ASVS v4.1 and a bit different discussion.

[tghosth]

I am going to push this to 4.1 as I think it is going to be too involved for 4.0.2

[ghost]

I have misgivings about AV scanning being mandated also. Or to be precise, shifting any responsibility for scanning of files from the enduser (uploader or downloader). I think the increased attack surface argument is valid. But also I think it is accepted that static analysis (likely to be what server-side solutions will perform) is just not effective. It is too easily bypassed - hence the rise in EPP and EDR solutions in enterprises. Consumer products are also usually much more robust than the AV engines typically found running against e.g S3 buckets.

I also think it puts us on a path towards discussing who is responsible for uploaded content more generally, as this is where this issue is covered in website terms and conditions. Uploading illegal images could be more damaging than malware to both the site owner and users. As it is for privacy requirements, a steer in the direction of legal advice feels more appropriate for applications hosting user supplied content.

[jmanico]

AV scanning is a very basic control that almost all of the big upload players support. This is a relatively simple and critical control to avoid a host is server and client side issues. I vote to keep it, but perhaps raise the risk category. **And remember the standard can be forked I want to have it in there to point it out but if your implementation disagrees go ahead fork the standard and remove this**

…

-- Jim Manico @manicode

On Aug 28, 2020, at 12:51 AM, garethwj72 ***@***.***> wrote:  I have misgivings about AV scanning being mandated also. Or to be precise, shifting any responsibility for scanning of files from the enduser (uploader or downloader). I think the increased attack surface argument is valid. But also I think it is accepted that static analysis (likely to be what server-side solutions will perform) is just not effective. It is too easily bypassed - hence the rise in EPP and EDR solutions in enterprises. Consumer products are also usually much more robust than the AV engines typically found running against e.g S3 buckets. I also think it puts us on a path towards discussing who is responsible for uploaded content more generally, as this is where this issue is covered in website terms and conditions. Uploading illegal images could be more damaging than malware to both the site owner and users. As it is for privacy requirements, a steer in the direction of legal advice feels more appropriate for applications hosting user supplied content. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[elarlang]

I use covid virus as metaphor...

So, @garethwj72 , you want to say, cashier in a shop (server, providing service for a lot of customers) don't need to wear a mask because it's shop visitor responsibility and problem, if they'll get a virus and only they need to wear masks?

People are willing to download content from trusted domains/sites, it completely to make sense to give some trust to this content on the server side. Client side hygiene is different topic and outside of ASVS scope.

And with this attitude, we should not have anti-virus checks, spam filters etc on email servers and hope that end-user can handle it.

[jmanico]

When you do anti-virus on download in addition to upload, you’re also protecting clients who download files. It’s a good idea. I typically flag a downloaded file with the antivirus version number in my database so I only have to do antivirus on download once per antivirus signature type, or only once per day for better performance. By all means don’t do this if you don’t want to but keeping this requirement in is my file upload Alamo and it’s critical to Security. :)

…

-- Jim Manico @manicode

On Aug 28, 2020, at 7:44 AM, Elar Lang ***@***.***> wrote:  I use covid virus it as metaphor... So, @garethwj72 , you want to say, cashier in a shop (server, providing service for a lot of customers) don't need to wear a mask because it's shop visitor responsibility and problem, if they'll get a virus and only they need to wear masks? People are willing to download content from trusted domains/sites, it completely to make sense to give some trust to this content on the server side. Client side hygiene is different topic and outside of ASVS scope. — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[ghost]

@elarlang - without resorting to metaphors, I have yet to see a web application that does scanning for malware that isn't easy to bypass. Your average office worker has figured out they can sneak things around using zip+password. The average pentest goes as far as throwing an Eicar string at it. I just think it's a weak control that's too vague and too tied up in a whole other area of policing user created content.

@jmanico Just adding to the discussion as Daniel requested above, with no expectation of it being removed. I adapt ASVS to my needs and there is very little in it I won't stand behind.

[ay-kay]

I disagree with @jmanico that all big players do this, during upload or in general. Microsoft claims to scan asynchronously in the background and to warn or block when downloading. Google states to scan for viruses before a file is downloaded. Users can download the virus-infected file, but only after acknowledging the risk of doing so. Note that Google also limits AV scanning to files smaller than 100 MB. Dropbox does not do AV scanning at all and recommends client-side scanning. iCloud does not seem to do any AV scanning - but there are no viruses on Macs either 😉

I also see some disadvantages with the scan on upload approach. If you can provoke a denial of service for the whole application with just an Eicar upload, this might not be good. Can we inform users about such difficulties? I also see the point that signatures may be more up-to-date at the time of download. I also understand @elarlang's argument, that special attention must be payed to public upload folders. But as @garethwj72 points out, there are not only viruses, which can cause damage.

In the end, we must also investigate whether traditional signature-based AV scanning is still effective and relevant today, whether on the client or on the server-side.

All in all, I think that already the discussion here shows that this requirement can be improved.

Finally, I really don't like the idea that everyone should fork their own ASVS. This is not my understanding on how to move a common standard forward.

[jmanico]

Finally, I really don't like the idea that everyone should fork their own ASVS. This is not my understanding on how to move a common standard forward.

This is described in detail in the primary document to explain why. ASVS is not a typical standard. - Jim

…

On 8/28/20 9:28 AM, ay-kay wrote: I disagree with @jmanico <https://github.com/jmanico> that all big players do this. Microsoft <https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/virus-detection-in-spo?view=o365-worldwide> claims to scan asynchronously in the background and to warn or block when downloading. Google <https://support.google.com/a/answer/172541?hl=en> states to scan for viruses before a file is downloaded. Users can download the virus-infected file, but only after acknowledging the risk of doing so. Note that Google also limits AV scanning to files smaller than 100 MB. Dropbox <https://help.dropbox.com/en-en/accounts-billing/security/viruses-malware> does not do AV scanning at all and recommends client-side scanning. iCloud does not seem to do any AV scanning - but there are no viruses on Macs either 😉 I also see some disadvantages with the scan on upload approach. If you can provoke a denial of service for the whole application with just an Eicar upload, this might not be good. Can we inform users about such difficulties? I also see the point that signatures may be more up-to-date at the time of download. I also understand @elarlang <https://github.com/elarlang>'s argument, that special attention must be payed to public upload folders. But as @garethwj72 <https://github.com/garethwj72> points out, there are not only viruses, which can cause damage. In the end, we must also investigate whether /traditional signature-based/ AV scanning is still effective and relevant today, whether on the client or on the server-side. All in all, I think that already the discussion here shows that this requirement can be improved. Finally, I really don't like the idea that everyone should fork their own ASVS. This is not my understanding on how to move a common standard forward. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#679 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEBYCLS5O7DNDKP2RCYHWTSDAAO5ANCNFSM4I3EX4KA>.

[elarlang]

One more idea or point-of-view - we don't need to put to the requirement, what is the world-wide situation at the moment (lack of virus checks), we need to set the goal or direction, where we would like to go.

[jmanico]

I feel like your comment is proving my point. Lots of the major players do indeed do anti-virus as part of their upload/download solution. I am happy to wordsmith the requirement so long as it is added somehow. - Jim

…

On 8/28/20 9:28 AM, ay-kay wrote: I disagree with @jmanico <https://github.com/jmanico> that all big players do this. Microsoft <https://docs.microsoft.com/en-us/microsoft-365/security/office-365-security/virus-detection-in-spo?view=o365-worldwide> claims to scan asynchronously in the background and to warn or block when downloading. Google <https://support.google.com/a/answer/172541?hl=en> states to scan for viruses before a file is downloaded. Users can download the virus-infected file, but only after acknowledging the risk of doing so. Note that Google also limits AV scanning to files smaller than 100 MB. Dropbox <https://help.dropbox.com/en-en/accounts-billing/security/viruses-malware> does not do AV scanning at all and recommends client-side scanning. iCloud does not seem to do any AV scanning - but there are no viruses on Macs either 😉 I also see some disadvantages with the scan on upload approach. If you can provoke a denial of service for the whole application with just an Eicar upload, this might not be good. Can we inform users about such difficulties? I also see the point that signatures may be more up-to-date at the time of download. I also understand @elarlang <https://github.com/elarlang>'s argument, that special attention must be payed to public upload folders. But as @garethwj72 <https://github.com/garethwj72> points out, there are not only viruses, which can cause damage. In the end, we must also investigate whether /traditional signature-based/ AV scanning is still effective and relevant today, whether on the client or on the server-side. All in all, I think that already the discussion here shows that this requirement can be improved. Finally, I really don't like the idea that everyone should fork their own ASVS. This is not my understanding on how to move a common standard forward. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#679 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEBYCLS5O7DNDKP2RCYHWTSDAAO5ANCNFSM4I3EX4KA>.

[jmanico]

Commit 8b3363c takes on suggestions of Elar and I'm closing this out. I regret we are not removing this per the original posters request and suggest those who disagree to fork the standard and remove this requirement for their teams.

PS: AntiVirus on upload is much safer in a serverless function.