File Upload - scan in-memory or after saving

Question

The web-application I'm working on will allow users to upload files (Word / PDF etc.). I intend to run a clamav scan on the files after upload before anything further is done with them.

From a security perspective is there any diference or advantage to scanning them in the web-server as soon as received (i.e. Node.JS has the file in memory and send it into Clam using clamscan npm library); versus saving the file and then scanning it (e.g. save it to S3 and have a Lambda trigger a Clam scan of the file).

I'm leaning towards the former as it then scans it as soon as is possible, but am interested on if there are notable security factors for one option or the other. Files are less than 10MB in size and we plan to take other precautions such as only accepting certain file types, checking the file type, limiting the size of the filename etc.

Answer 1

Scanning uploaded files in memory before they reach the file system is not the general option. Of course HTTP allows to limit the upload size of a file, but a few concurrent upload will easily consume all the available memory and crash the application unless it is specially designed to be protected against that.

The more common way is to upload files in a temporary zone acting like a spool:

• when a file has been fully uploaded it is queued to be scanned
• an automatic engine reads that queue and executes the anti-malware system on the file
• only if nothing is detected the file goes into its final destination. Else the file is either immediately destroyed or falls in a quarantine zone for further analysis.

That way you do not need memory to keep everything in memory, yet can be sure that only clean files (according to your anti-malware...) can be made available.

Answer 2

If your goal is to scan the files before they hit the filesystem, scanning the data in memory after the web server handles the upload from a HTTP form is unlikely to resolve that, because almost every web server implements file uploads as streaming writes to a temporary file anyway. Buffering the whole file in memory is wasteful, and quickly becomes a recipe for an out-of-memory denial of service (OOM DoS) attack.

If you're receiving a file from the client via some custom transport (e.g. over a websocket) you should consider the same approach - receive a chunk, write it to a temporary file, keep your in-memory buffer small. While 10MB doesn't sound like a lot, an attacker only has to initiate couple of thousand uploads simultaneously to completely knock your web server offline.

Malware only becomes malicious if you actually execute it, or give the attacker some way of causing it to be executed. As such, as long as you place it somewhere where it won't be accessed in such a way (i.e. out of the webroot and not in a location where the OS will try to treat it is a script or config file) and ensure that the upload is secure against path traversal and other shenanigans, there's no real risk of you executing the file and having your server compromised.

The only other thing you should try to be careful of is an attacker using your server to host the malware. This means being careful not to allow the uploaded temporary files to be accessible via the web server, and not to allow unauthenticated access to files on the S3 bucket if you're using one.

You should also ensure that temporary files are deleted if the upload is aborted or some other failure occurs during the upload. Otherwise an attacker can start uploading a 1MB file, claim that it is 8MB, then terminate the upload after the file contents are uploaded. Since the upload didn't complete, this will likely skip your "scan with ClamAV after upload" trigger, and leave the file on disk in the temporary directory. Proper error handling can prevent this.