Converting a 3GB file into a byte array

Question

I am trying to convert 3GB file into byte array but getting OOM(OutOfMemoryError).

We tried

RandomAccessFile randomAccessFile = new RandomAccessFile(sourceLocation.getAbsolutePath(), "r");
MappedByteBuffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length()); //while it is large file, it threw 'mmap failed: ENOMEM (Out of memory)' exception.

byte[] data = new byte[1024 * 1024];  // 1MB read at time
while (mappedByteBuffer.hasRemaining()) {
    int remaining = data.length;

    if (mappedByteBuffer.remaining() < remaining)
        remaining = mappedByteBuffer.remaining();
        mappedByteBuffer.get(data, 0, remaining);
    }

    mappedByteBuffer.rewind();
    byte fileContent[] = mappedByteBuffer.array(); //while it is small file, it threw 'java.nio.ReadOnlyBufferException' exception.
    randomAccessFile.close();
}

My Custom Request Body: Custom request body class where my request body prepared

import android.os.Looper;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okio.BufferedSink;

public class ProgressRequestBodyTemp extends RequestBody
{
    private static final int DEFAULT_BUFFER_SIZE = 2048;
    private File mFile;
    private String mPath;
    private String mFileType;
    private int mItemIndex;
    private UploadCallbacks mListener;

    private byte[] encryptedData;
    private ByteArrayInputStream bis;

    public ProgressRequestBodyTemp(final String fileType, final File file, byte[] encryptedData, final int itemIndex, final UploadCallbacks listener)
    {
        this.mFile = file;
        this.mFileType = fileType;
        this.mItemIndex = itemIndex;
        this.mListener = listener;

        this.encryptedData = encryptedData;

        try
        {
            bis = new ByteArrayInputStream(encryptedData); // Convert byte array into input stream for send data to server
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }

    @Override
    public MediaType contentType()
    {
        try
        {
            return MediaType.parse(mFileType);
        }
        catch (Exception ex)
        {
            return MediaType.parse("");
        }
    }

    @Override
    public long contentLength() throws IOException
    {
        return encryptedData.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException
    {
        long fileLength = mFile.length();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];

        long uploaded = 0;
        try
        {
            int read;
            android.os.Handler handler = new android.os.Handler(Looper.getMainLooper());
            while ((read = bis.read(buffer)) != -1)
            {

                // update progress on UI thread
                handler.post(new ProgressUpdater(uploaded, encryptedData.length));

                uploaded += read;
                sink.write(buffer, 0, read);
            }
        }
        finally
        {
            bis.close();
        }
    }

    public interface UploadCallbacks
    {
        void onProgressUpdate(int itemIndex, int percentage);
    }

    private class ProgressUpdater implements Runnable
    {
        private long mUploaded;
        private long mTotal;

        public ProgressUpdater(long uploaded, long total)
        {
            mUploaded = uploaded;
            mTotal = total;
        }

        @Override
        public void run()
        {
            if (mListener != null)
            {
                mListener.onProgressUpdate(mItemIndex, (int) (100 * mUploaded / mTotal));
            }
        }
    }
}

My Request: request for upload file

File sourceLocation = new File(".....");
byte fileContent[] = .... // byte array

ProgressRequestBody requestFile  = new ProgressRequestBody(fileType, sourceLocation, fileContent, itemIndex, this);
MultipartBody.Part body = MultipartBody.Part.createFormData("file", sourceLocation.getName(), requestFile); // MultipartBody.Part is used to send also the actual file name
Call<String> mediaCall = getRetrofit().addMedia("SOME STRING DATA", body);

It is giving OutOfMemoryError, Please suggest the best way to convert the large file into byte array.

any help appreciated a lot. thanks in advance.

reading a whole 3 gb file into a memory is 100% bad idea. Why: At least not all devices have 3 gb of RAM, and even the ones which have it will never have all it free. Now if you have 3 GB of free RAM - VM won't let you take them. as you might be not the only one who wants it. So don't try to load whole 3gb. You will need to load and process it part by part — Vladyslav Matviienko
– Vladyslav Matviienko, Commented Jan 9, 2019 at 13:07
no we are using retrofit, in it we are using custom RequestBody @pskink — Ashish Pandya
– Ashish Pandya, Commented Jan 9, 2019 at 13:12
then how can I upload 3GB file, my api using byte array @VladyslavMatviienko — Ashish Pandya
– Ashish Pandya, Commented Jan 9, 2019 at 13:13

Daniel B. · Accepted Answer · 2019-01-09 13:42:59Z

Use InputStream and OutputStream.

They are meant for big amounts of data, for example, when you work with 3GB of data and can't load it into memory.

If you are trying to upload a file, use FileInputStream. Create a File object, pass it to the FileOutputStreamconstructor and start reading bytes from its InputStream to a byte[] buffer and send the bytes with an OutputStream to the server.

This approach will not cause an OutOfMemoryError,because you are reading only enough bytes to fill up the buffer, which should be about 2KB - 8KB in size. After the buffer is full, you write the bytes to the server. After the buffer was written to the server, you read into the buffer again, and the process keeps going on until the whole file is uploaded.

Example using FileInputStream

        File file = new File("yourfile.txt");
        FileInputStream fis = null;
        OutputStream outputStream = null;

        url = new URL(desiredUrl);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();


        try {
            fis = new FileInputStream(file);
            connection.setDoOutput(true);
            outputStream = connection.getOutputStream();

            int actuallyRead;
            byte[] buffer = new byte[2048];
            while ((actuallyRead = fis.read(buffer)) != -1) {
                //do something with bytes, for example, write to the server
            outputStream.write(buffer);

            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fis != null)
                    fis.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            try {
                if (outputStream != null)
                    outputStream.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }

        }

Note: This approach does not mean you need to reconnect to the server each time a buffer is filled. It will continuously write to the server, until it is done processing the file. This will happen all under the same, single connection.

my API allows the single call for a file, in the above method each time required to call upload service?
@AshishPandya No. you establish a connection with the server, get the OutputStream object from the connection, and keep writing to it until you are done. This all happens in 1 connection, no multiple re-connections.

Collectives™ on Stack Overflow

Converting a 3GB file into a byte array

1 Answer 1

Example using FileInputStream

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Example using FileInputStream

3 Comments

Related