1

I am trying to read a extremely large file to an array. it is obvious that i am not reading it to an array at the moment but that is not the problem i am having. it is about 600 million digits and this seems to cause it to get an error. Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

BufferedReader brWorld = null;
String World;
brWorld = new BufferedReader(new FileReader("Saves/" + Name));
World = brWorld.readLine();
String Numbers = World;
String[] LoadWorld = Numbers.split("");
for(int x = 0; x < MaterialArray.length * MaterialArray.length; x++){
    System.out.println(String.valueOf(LoadWorld[x]));
}

Im Loading a would into an array for my game. Is there any way of loading a text file this big? thanks Liam

5
  • 1
    Only was is increase RAM, if you really need entire file in memory. Otherwise read chunk by chunk Commented Apr 17, 2014 at 19:17
  • Please do NOT name variables with upper case first letter, there is a strong java convention to start variables with lower case letters, and it makes your code unreadable for java developers Commented Apr 17, 2014 at 19:18
  • String[] LoadWorld = Numbers.split(""); Did you really mean an empty string there? what are you trying to achieve? Commented Apr 17, 2014 at 19:19
  • i know the code works. it just stops working when the file is to big Commented Apr 17, 2014 at 19:30
  • @user3543533 see my solution to your problem. You should be using string.charAt() instead of breaking the string into an array. Commented Apr 17, 2014 at 19:36

2 Answers 2

3

First off, you'll need a lot of memory for this. Several GBs of RAM, not including GC room.

If you're reading a "single line" of 600M digits, in the end, after the line is actually read (you're likely not even getting this far), the string of 600M digits will require 1.2GB of memory, simply for the characters.

Java stores Strings as arrays of Characters, and Characters are stored internally as UTF-16, which is 2 bytes.

That alone sends the memory requirements through the roof.

When you do your split, you're turning each digit in to an individual string. You now have 600M Strings, and all of the over head that entails. At a minimum, you're looking at least 16 bytes per string, since it stores the pointer to the underlying array, an offset in to the array, and the length of the string. Thankfully, split will actually reuse the underlying array, but that's really not much help here.

16 bytes * 600M is 9.6GB.

Definitely heading in to the "ludicrous" realm of memory requirements now.

You don't say what you actually want to do. You want to "load the file", but you don't say in to what. So, it's hard to provide a recommendation as to how it should be done. If you want to just print the file out, then you can simply read each character one by one and print it, but that's clearly not the goal.

So, while, yes, you can "throw memory" at this problem, you don't want to. You need to study the problem and come up with a better representation of what you're trying to achieve and work from there.

Sign up to request clarification or add additional context in comments.

Comments

0

Here is why you are running out of memory:

brWorld = new BufferedReader(new FileReader("Saves/" + Name));
World = brWorld.readLine();
String Numbers = World;
String[] LoadWorld = Numbers.split("");

You are first reading in all of the digits:

600,000,000 chars = 600,000,000 bytes = 600 MB

So already you are at 600MB at least. Then you are creating a whole new array of Strings. This means that you are creating a String pointer for each char in the array. Based on your architecture, this could be 4 bytes or more:

600 MB + 600,000,000 * 4 = 600 MB + 2,400 MB = 3,000 MB = 3 GB

*These numbers are estimates and I know the actual amount is probably MUCH larger.

As you can see, this is not a feasible solution at all. This is a MUCH better solution:

BufferedReader brWorld = null;
String world;
brWorld = new BufferedReader(new FileReader("Saves/" + Name));
world = brWorld.readLine();
for(int x = 0; x < MaterialArray.length * MaterialArray.length; x++){
    System.out.println(String.valueOf(world.charAt(x)));
}

1 Comment

This is way over 600MB. He is creating 600,000,000 String objects, each weights much more than 1B...

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.