Reading bytes from large files in Java cause Java heap space error

Question

Awhile ago I was seeking help on the code below and finally started working on it again. Basically, I've narrowed my error down to the size of the file causing this error:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

The line right below that error in the stack trace is: at java.util.Arrays.copyOf(Arrays.java:2786)

I can pass this program a large directory with thousands of smaller files, but any file over this 50 Mb size tends to crash. I haven't traced the exact size that the program crashes on but I know that at least a 50 Mb file causes issues.

Below are the primary snippets and where the stack trace tells my that my code is breaking.

private void handleFile(File source)
{
    FileInputStream fis = null;

    try
    {
        if(source.isFile())
        {
            fis = new FileInputStream(source);
            handleFile(source.getAbsolutePath(), fis);
        }
        else if(source.isDirectory())
        {
            for(File file:source.listFiles())
            {
               if(file.isFile())
               {
                   fis = new FileInputStream(file);
                   handleFile(file, fis);
               }
               else
               {
                   handleFile(file);
               }
            }
         }
     }
     catch(IOException ioe)
     {
         ioe.printStackTrace();
     }
     finally
     {
         try
         {
             if(fis != null) { fis.close(); }
         }
         catch(IOException ioe) { ioe.printStackTrace(); }
     }
}

private handleFile(String fileName, InputStream inputStream)
{
    byte[] startingBytes = null;

    try
    {
       startingBytes = inputStreamToByteArray(inputStream);

       if(startingBytes.length == 0) return;

       if(isBytesTypeB(startingBytes))
       {
          do stuff
          return;
       }
     }
     catch(IOException ioe)
     {
         ioe.printStackTrace();
     }
}

private byte[] inputStreamToByteArray(InputStream inputStream)
{
    BufferedInputStream bis = null;
    ByteArrayOutputStream baos = null;

    try
    {
        bis = new BufferedInputStream(inputStream);
        baos = new ByteArrayOutputStream(bis);

        byte[] buffer = new byte[1024];

        int nRead;
        while((nRead = bis.read(buffer)) != -1)
        {
            baos.write(buffer, 0, nRead);
        }
    }
    finally { baos.close(); }

    return baos.toByteArray();
 }

 private boolean isBytesTypeB(byte[] fileBytes)
 {
     // Checks if these bytes match a particular type
     if(BytesMatcher.matches(fileBytes, fileBytes.length))
     {
         return true;
     }
     return false;
 }

So there is something in the above code that is causing the error. Any ideas what I'm doing wrong here?

Answer 1

Arrays.copyOf is invoked each time the ByteArrayOutputStream's internal array needs resizing. This is the moment of the highest memory demand. You can avoid array resizing by specifying the initial size of the array equal to the file size.

Answer 2

I haven't read all your code, but it is possible to start Java with more heap space available

java -Xmx128m

for example.

Answer 3

you can increase your heap space from Windows> Preferences> Java> Installed JREs from there select the JRE and click edit, then write in the Default VM Arguments: to -Xmx2048 (it will allocate 2gb)