Android OutOfMemoryError when retrieving large amount of data

Question

I have an app that makes a web call and retrieves XML data. The code below works fine if there is not too much data.

public class WebClient {

    private static final String TAG = WebClient.class.getSimpleName();
    private String result;


    private static String convertStreamToString(InputStream is) {
        /*
         * To convert the InputStream to String we use the
         * BufferedReader.readLine() method. We iterate until the BufferedReader
         * return null which means there's no more data to read. Each line will
         * appended to a StringBuilder and returned as String.
         */
        BufferedReader reader = new BufferedReader(new InputStreamReader(is));
        StringBuilder sb = new StringBuilder();

        String line = null;
        try {
            while ((line = reader.readLine()) != null) {
                sb.append(line + "\n");
            }
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                is.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return sb.toString();
    }


    public String connect(String url) {
        Log.e(TAG, "inside LoginWebClient.connect(url). url = " + url);

        result = null;

        HttpClient httpclient = new DefaultHttpClient();

        // Prepare a request object
        HttpGet httpget = new HttpGet(url);


        // Execute the request
        HttpResponse response;
        try { 
            response = httpclient.execute(httpget);
            // Examine the response status
            Log.i(TAG, response.getStatusLine().toString());





            // Get hold of the response entity
            HttpEntity entity = response.getEntity();
            // If the response does not enclose an entity, there is no need
            // to worry about connection release

            if (entity != null) {


                InputStream instream = entity.getContent();
                result = convertStreamToString(instream);
                //Log.i(TAG, result);



                // Closing the input stream will trigger connection release
                instream.close();

            }

                } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        Log.e(TAG, "inside WebClient.connect(url). result = " + result);

         return result;

    }//end of connect

.

If the app makes a web call that returns a lot of data, i get a OutOfMemoryError. I've googled the error and it can be caused by holding too much data in memory in the StringBuilder.

As a workaround i have read i can read the Stream straight into a String to avoid the StringBuilder.

i imported the com.google.common.io.CharStreams jar file into my project and tried the following:

private static String convertStreamToStringForGetAlerts(InputStream is) {

        String stringFromStream = null;


        try {
            stringFromStream = CharStreams.toString(new InputStreamReader(is, "UTF-8"));
        } catch (UnsupportedEncodingException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        return stringFromStream;
    }

But i still get the following error.

06-05 09:44:40.809: E/AndroidRuntime(5851): FATAL EXCEPTION: AsyncTask #2
06-05 09:44:40.809: E/AndroidRuntime(5851): java.lang.RuntimeException: An error occured while executing doInBackground()
06-05 09:44:40.809: E/AndroidRuntime(5851):     at android.os.AsyncTask$3.done(AsyncTask.java:278)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.FutureTask$Sync.innerSetException(FutureTask.java:273)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.FutureTask.setException(FutureTask.java:124)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:307)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.FutureTask.run(FutureTask.java:137)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1076)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:569)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.lang.Thread.run(Thread.java:856)
06-05 09:44:40.809: E/AndroidRuntime(5851): Caused by: java.lang.OutOfMemoryError
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.lang.AbstractStringBuilder.enlargeBuffer(AbstractStringBuilder.java:94)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.lang.AbstractStringBuilder.append0(AbstractStringBuilder.java:162)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.lang.StringBuilder.append(StringBuilder.java:311)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.lang.StringBuilder.append(StringBuilder.java:44)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.google.common.io.CharStreams.copy(CharStreams.java:204)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.google.common.io.CharStreams.toStringBuilder(CharStreams.java:245)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.google.common.io.CharStreams.toString(CharStreams.java:219)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.carefreegroup.rr3.carefreeoncall.WebClient.convertStreamToStringForGetAlerts(WebClient.java:151)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.carefreegroup.rr3.carefreeoncall.WebClient.connectForGetAlerts(WebClient.java:198)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.carefreegroup.rr3.carefreeoncall.WebService.getAlerts(WebService.java:1778)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.carefreegroup.rr3.carefreeoncall.ShowAlertsActivity$AsyncGetAlerts.doInBackground(ShowAlertsActivity.java:88)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at com.carefreegroup.rr3.carefreeoncall.ShowAlertsActivity$AsyncGetAlerts.doInBackground(ShowAlertsActivity.java:1)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at android.os.AsyncTask$2.call(AsyncTask.java:264)
06-05 09:44:40.809: E/AndroidRuntime(5851):     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
06-05 09:44:40.809: E/AndroidRuntime(5851):     ... 4 more

Has anyone any ideas how i can read this Stream into a String without causing this Error?

Thanks in advance.

Stephen C · Accepted Answer · 2014-06-05 09:16:39Z

As a workaround i have read i can read the Stream straight into a String to avoid the StringBuilder.

That doesn't help. Underneath the covers, the CharStreams method is most likely using a StringBuilder or equivalent ... and since it cannot know how big the builder needs to be, it will be using the builder's "double-the-size" strategy to expand the backing array.

The most memory efficient way to read data into memory would be to preallocate some kind of buffer (a StringBuilder, a byte[], a char[], etc) with a size that is big enough to hold the characters / bytes, and then fill the buffer. You will still need twice the amount of memory if you need to turn the buffer contents into a String.

But this is only staving off the inevitable scaling problem. Eventually, the XML will be too big to hold in memory. What you really need to do is to change your code to not try to hold the entire XML document in memory. You could do one of the following:

feed it to an event driven XML parser (e.g. SAX) to extract the information of interest, or
write it to a local file, a byte, character, line or buffer at a time.

Juan Pedro Martinez · Accepted Answer · 2014-06-05 09:04:40Z

0

I think the issue is in the new InputStreamReader (....) . why you don't try to apply the byte conversion in the middle of the process.

public byte[] read(InputStream istream)
{
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   byte[] buffer = new byte[1024]; // Experiment with this value
   int bytesRead;

   while ((bytesRead = istream.read(buffer)) != -1)
   {
      baos.write(buffer, 0, bytesRead);
   }

   return baos.toByteArray();
}


// after the process is run, we call this method with the String
public void readLines(byte[] data)
{
  BufferedReader reader = new BufferedReader(new InputStreamReader(new ByteArrayInputStream(data)));
  String line;

  while ((line = reader.readLine()) != null)
  {
    // do stuff with line
  }
}

answered Jun 5, 2014 at 9:04

Juan Pedro Martinez

1,9741 gold badge15 silver badges24 bronze badges

4 Comments

turtleboy Over a year ago

It doesn't crash now but i cannot get the xml string. All that is returned is the address of the byte[] object eg 06-05 10:47:57.127: E/WebClient(6102): inside WebClient.connect(url) byteArray.toString() = [B@411bc420

Juan Pedro Martinez Over a year ago

byte[] bytes = {...} String str = new String(bytes, "UTF-8"); // for UTF-8 encoding

Juan Pedro Martinez Over a year ago

I think that way is the proper one...hope to help you!

turtleboy Over a year ago

sorry i get the OutOfMemory error. i think i might have to write the response to disk.

JafarKhQ · Accepted Answer · 2014-06-05 09:07:00Z

0

Are you sure your data is that large ? what is the size of the XML ?
anyway, You have to cash your data on the disc (preferences, sqlite or file), then read or parse the cached data chunk by chunk.

answered Jun 5, 2014 at 9:07

JafarKhQ

8,7343 gold badges37 silver badges45 bronze badges

Comments

iamreza · Accepted Answer · 2015-07-02 21:27:47Z

0

You can either compress data using a deflater and inflater , or send strings from server in lines using \n and a printwriter , then write the data to a temporary file on the clinet, line by line

answered Jul 2, 2015 at 21:27

iamreza

5710 bronze badges

Collectives™ on Stack Overflow

Android OutOfMemoryError when retrieving large amount of data

4 Answers 4

Comments

4 Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

4 Comments

Comments

Comments

Related