Java Heap Space Error while reading large files

Question

I am trying to load an entire file in memory as a string object.But after the application is over,my memory is not getting released back to garbage collector.I know reading the entire file in memory is bad,bu I have to send the data to another class,could anyone help me how to do this as just a stream instead of loading the entire code in memory,if not,could anyone say what is wrong with my present code

private   String processFile(FileItemStream item) throws IOException{

        InputStream is=null;

        try{
            is=item.openStream();
            return IOUtils.toString(is, "UTF-8");
        }
        finally {

            IOUtils.closeQuietly(is);

        }

    }




data=processFile(item)//method call
SomeClass(data);//passing the data string to this class

Answer 1

You should pass your is into the class that needs the data. As long as you don't read in all the contents of is at once, you won't use up all your memory. Change your code to this:

InputStream is = null;
try {
    is = getFileAsStream(item);
    SomeClass(is)  //of course this probably doesn't compile, I'm just using your code sample
} finally {
    //close is
}

Answer 2

It depends on the API that is available to you. If their API lets you process a byte array of data at a time then you should read a chunk into a buffer and send it to that application. If it lets you pass in an InputStream then you should do that.

If the API does not allow you to do one of the above then the only other thing you can is increase the amount of memory that Java can use. To do this start Java with the -Xmx512m switch where 512 is the amount of memory you can allocate to Java.

Answer 3

If you really can't change the API that you are calling to take something besides a string, then you will need to work around the problem by giving your JVM more memory. This can be changed by sending a parameter to the JVM.

-Xmx<size>

For example you can use a setting like this to give you JVM a max heap of 2GB

-Xmx2g

Though if you are using an API that accepts a String as a parameter and are trying to pass a multi-gigabyte string you are almost certainly using their API incorrectly. If someone builds an API that expects large amounts of data they would not build it with a single string as a parameter. Good luck.