2

I am developing a File Uploader for big file. Upload from HTML script and send by byte from Javascript using ArrayBuffer and Unit8Array to PHP. The PHP script will stream the file and save it into folder.

Here's my Javascript looks like

function upload(fileInputId, fileIndex)
    {
        var file = document.getElementById(fileInputId).files[fileIndex];
        var blob;
        var reader = new FileReader();
        reader.readAsBinaryString(file); 
        reader.onloadend  = function(evt)
        {
            xhr = new XMLHttpRequest();

            xhr.open("POST", 'upload.php?name=' + file.name, true);

            XMLHttpRequest.prototype.mySendAsBinary = function(text){
                var data = new ArrayBuffer(text.length);
                var ui8a = new Uint8Array(data, 0);
                for (var i = 0; i < text.length; i++){ 
                    ui8a[i] = (text.charCodeAt(i) & 0xff);

                }

                if(typeof window.Blob == "function")
                {
                     blob = new Blob([data]);
                }else{
                     var bb = new (window.MozBlobBuilder || window.WebKitBlobBuilder || window.BlobBuilder)();
                     bb.append(data);
                     blob = bb.getBlob();
                }

                this.send(blob);
            }

            var eventSource = xhr.upload || xhr;
            eventSource.addEventListener("progress", function(e) {
                var position = e.position || e.loaded;
                var total = e.totalSize || e.total;
                var percentage = Math.round((position/total)*100);
            });

            xhr.onreadystatechange = function()
            {
                if(xhr.readyState == 4)
                {
                    if(xhr.status == 200)
                    {
                        console.log("Done");
                    }else{
                        console.log("Fail");
                    }
                }


            };

            xhr.mySendAsBinary(evt.target.result);
        };
    }

This is my upload.php

$inputHandler = fopen('php://input', "r");
$loc = "uploads/" . $_GET["name"];
$fileHandler = fopen($loc, "w+");

while(true) {
    $buffer = fgets($inputHandler, 4096);



    if (strlen($buffer) == 0) {
        fclose($inputHandler);
        fclose($fileHandler);
        return true;
    }

    fwrite($fileHandler, $buffer);
}

My question is, how do I encrypt those upload file using AES or mcrypt while the file is in streaming mode?

9
  • 1
    Why would you want to, I sort of thought that was what https SSL/TLS was for. Also depending how big it is, AES will eventually barf on you. Maybe 50-100MB. You can do something like I did. Take about 1MB, encrypt it, add a : take another 1MB encrypt it. etc. Then when you decrypt, you just read the file tell you get to a : decode a chunk, etc. etc. Basically encrypt it one chunk at a time. Commented Mar 30, 2018 at 2:00
  • I mean is, the file will stored as encrypted file. maybe looks like image.enc or something like that Commented Mar 30, 2018 at 2:03
  • How big is it? You will hit a memory wall, sooner or later trying to encrypt a large file. I forget how big we got up to, but we have 54GB of ram and had to encrypt chunk by chunk. Commented Mar 30, 2018 at 2:04
  • I would also avoid mycrypt - This feature was DEPRECATED in PHP 7.1.0, and REMOVED in PHP 7.2.0. I just had to re-code my AES, because were moving to PHP7 in a few months. And the encryption is not backwards compatible, meaning I had to re-ecrypt everything. So I decided to put the big file runs out of memory issue to bed. Commented Mar 30, 2018 at 2:06
  • On our system we put maximum as 1GB only. Encrypt by MB make sense Commented Mar 30, 2018 at 2:08

1 Answer 1

3

It was something like this. This is from memory and untested, because I don't have the PHPSecLib library on my Laptop, and I am too lazy to set that all up...

require __DIR__ . '/vendor/autoload.php';

use phpseclib\Crypt\AES;
use phpseclib\Crypt\Random;

AESStreamEncode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);
    
    $iv = Random::string($cipher->getBlockLength() >> 3);
    $cipher->setIV($iv);
    
    $base64_iv = rtrim(base64_encode($iv), '='); //22 chars
    
    fwrite($output, $base64_iv); //store the IV this is like a salt

    while(!feof($input)) {
        $contents = fread($input, 1000000); //number of bytes to encrypt 
        $encrypted = $cipher->encrypt($contents);
        //trim the = or ==, and replace with :, write to output stream.
        fwrite($output, rtrim(base64_encode($encrypted), '=').':'); 
    }
}

AESStreamDecode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);
    
    $buffer = '';
    $iv = false;
    
    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char ==':'){
            if(!$iv){
                $iv = base64_decode(substr($buffer, 0, 22).'=');  //iv is the first 22 of the first chunk.
                $cipher->setIV($iv);
                $buffer = substr($buffer, 22); //remove the iv
            }
            $buffer = base64_decode($buffer.'='); //decode base64 to bin
            $decrypted = $cipher->decrypt($buffer);
            fwrite($output, $decrypted);
            
            $buffer = ''; //clear buffer.
        }else{
            $buffer .= $char;
        }
    }
}

Where $input and $output are valid resource stream handles like from fopen etc.

 $input = fopen($filepath, 'r');
 $output = fopen($ohter_filepath, 'w');

 AESStreamEncode($input, $output, $key);

This lets you use things like php://output as the stream if downloading the decrypted file.

You have to remove the = because it is sometimes missing or 2 of them, So we cant rely on them as a separator. I usually just put 1 back on and it always decodes it correctly. I think it's just some padding anyway.

References

PHPSecLib on GitHub

PHPSecLib Examples

The encrypted file should look something like this:

xUg8L3AatsbvsGUaHLg6uYUDIpqv0xnZsimumv7j:zBzWUn3xqBt+k1XP0KmWoU8lyfFh1ege:nJzxnYF51VeMRZEeQDRl8:

But with longer chunks. The IV is like a salt and it's pretty common practice to just add it to the front or back of the encrypted string. So for example

[xUg8L3AatsbvsGU]aHLg6uYUDIpqv0xnZsimumv7j:

The part in the [] is the IV, (its 22 chars long after base64_encode) I counted it many times and it always comes out that long. We only need to record the IV and set it one time. I suppose you could do a different IV for each chunk, but whatever.

If you do use PHPSecLib, it also has some nice sFTP stuff in it. Just make sure to get the 2.0 version. Basically it has some fallbacks and native PHP implementations for different encryption algos. So like it would try open_ssl then if you were missing it, it would use their native implementation. I use it for sFTP, so I already had it available. sFTP requires an extension ssh2_sftp and If I recall it was only available on Linux at the time we set things up.

UPDATE

For downloading you can just issue the headers then give the decode function the output stream, something like this

 $input = fopen('encrypted_file.txt', 'r');
 $output = fopen('php://output', 'w');

 header('Content-Type: "text/plain"');
 header('Content-Disposition: attachment; filename="decoded.txt"');

 header('Expires: 0');
 header('Cache-Control: must-revalidate, post-check=0, pre-check=0, max-age=0');
 header("Content-Transfer-Encoding: binary");
 header('Pragma: public');

 //header('Content-Length: '.$fileSize);  //unknown

 AESStreamDecode($input, $output, $key);

These are pretty standard headers. The only real catch is because the filesize is different when it's encryped you can't just simply get the size of the file and use that as it will be quite a bit bigger. Not passing the filesize won't prevent the download, it just wont have an estimated time etc.

But because we know the size before encrypting it, we could embed it in the file data itself like this:

 3555543|xUg8L3AatsbvsGUaHLg6uYUDIpqv0xnZsimumv7j:zBzWUn3xqBt+k1XP0KmWoU8lyfFh1ege:nJzxnYF51VeMRZEeQDRl8:

And then pull it out when we do the download, but you would have to use as separate function to get it and it might be a bit tricky to not mess up decoding the file.

Honestly I think it's more hassle then it's worth.

UPDATE2

Anyway, I worked up these changes for embedding the file size, it's an option, but it could also mess up the decryption of the file if not done carefully. (I haven't tested this)

AESStreamEncode($input, $output, $key, $filesize = false)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);

    $iv = Random::string($cipher->getBlockLength() >> 3);
    $cipher->setIV($iv);

    $base64_iv = rtrim(base64_encode($iv), '='); //22 chars
    
    //Option1 - optional filesize
    if(false !== $filesize){
        //add filesize if given in the arguments
        fwrite($output, $filesize.'|');
    }
    
    /*
        //Option2: using fstat, remove '$filesize = false' from the arguments
        $stat = fstat($input);
        fwrite($output, $stat['size'].'|');
    */

    fwrite($output, $base64_iv); //store the IV this is like a salt

    while(!feof($input)) {
        $contents = fread($input, 1000000); //number of bytes to encrypt 
        $encrypted = $cipher->encrypt($contents);
        //trim the = or ==, and replace with :, write to output stream.
        fwrite($output, rtrim(base64_encode($encrypted), '=').':'); 
    }
}

So now we should have the filesize 3045345|asdaeASE:AEREA etc. Then we can pull it back out when decrypting.

AESStreamDecode($input, $output, $key)
{
    $cipher = new AES(AES::MODE_CBC);
    $cipher->setKey($key);

    $buffer = '';
    $iv = false;
    $filesize = null;

    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char =='|'){
            /*
              get the filesize from the file,
              this is a fallback method, so it wont affect the file if
              we don't pull it out with the other function (see below)
            */
            $filesize = $buffer;
            $buffer = '';
        }elseif($char ==':'){
            if(!$iv){
                $iv = base64_decode(substr($buffer, 0, 22).'=');  //iv is the first 22 of the first chunk.
                $cipher->setIV($iv);
                $buffer = substr($buffer, 22); //remove the iv
            }
            $buffer = base64_decode($buffer.'='); //decode base64 to bin
            $decrypted = $cipher->decrypt($buffer);
            fwrite($output, $decrypted);

            $buffer = ''; //clear buffer.
        }else{
            $buffer .= $char;
        }
    }
    //when we do a download we don't want to wait for this
    return $filesize;
}

The decode get filesize part acts as a fallback, or if you don't need it then you don't have to worry about it messing the file up when decoding it. When downloading we can use the following function, that way we don't have to wait for the file to be completely read to get the size (this is basically the same as what we did above).

//We have to use a separate function because
//we can't wait tell reading is complete to 
//return the filesize, it defeats the purpose
AESStreamGetSize($input){
    $buffer = '';
    //PHP_INT_MAX (maximum allowed integer) is 19 chars long
    //so by putting a limit of 20 in we can short cut reading
    //if we can't find the filesize
    $limit = 20;
    $i; //simple counter.
    while(!feof($input)) {
        $char = fgetc($input); //get a single char
        if($char =='|'){
            return $buffer;
        }elseif($i >= $limit){
            break;
        }
        $buffer .= $char;
        ++$i; //increment how many chars we have read
    }
    return false;
}

Then when downloading you just need to make a few changes.

$input = fopen('encrypted_file.txt', 'r');
//output streams dumps it directly to output, lets us handle larger files
$output = fopen('php://output', 'w');
//other headers go here

if(false !== ($filesize = AESStreamGetSize($input))){
    header('Content-Length: '.$fileSize);  //unknown
    //because it's a file pointer we can take advantage of that
    //and the decode function will start where the getSize left off.
    // or you could rewind it because of the fallback we have.
    AESStreamDecode($input, $output, $key);
}else{
    //if we can't find the filesize, then we can fallback to download without it
    //in this case we need to rewind the file
    rewind($input);
    AESStreamDecode($input, $output, $key);
}

If you want to shorten this you can just do it this way too, it's only about 19 chars at most so it's not to big a performance issue.

 if(false !== ($filesize = AESStreamGetSize($input))) header('Content-Length: '.$fileSize);

 rewind($input);
 AESStreamDecode($input, $output, $key);

Basically above, we just do the filesize header (or not) and then rewind and do the download. It will re-read the filesize, but that's pretty trivial.

For reference fstat(), Hopefully that makes sense.

Sign up to request clarification or add additional context in comments.

9 Comments

Thanks for the code. Btw, the AESStreamEncode should be in the while(true) code block or the outside? The input is the $inputHandler (which is I am using php://input) or $fileHandler (wrtten file)?
I've never done an upload like in your Question, so I don't know if it gives you the whole file or just some bytes at a time. This is just from memory ... lol. I wrote that like 2 weeks ago or so. I also didn't test is so hopefully there are not many errors.
I think in your case, I would replace the read loop while(!feof) with your while(true) then $input = $inputHandler and $output = $fileHandler, You also may have to loop a few times to get the amount of bytes you need, as I said I don't know if that is like an asynchronous stream, where you can only get so many bytes at a time. fgets reads only 1 line for example.
It may be possible to read your stream just like a normal file. As I mentioned above fgets only reads to the next \n new line or the byte limit. So you could check the length of buffer if you need to loop more then once to get the number of bytes you want. That is kind of a balance, you don't want it so long that it causes memory issues, but you also don't want it so short you call encrypt to many times.
The size of encrypted file are bigger 30-40% than original file. Is it normal?
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.