I've got a .txt with 1 Billion digits of pi. I read in the file as a String but i get an OutOfMemoryError. It worked with 1 MIllion digits tho. I save the String as a char[] array. Is it possible to somehow stream the .txt when i cycle through the whole array? I simply need a way to have access to all the 1 Billion digits as an array.
4 Answers
There is BufferedInputStream since java 1 or FileReader with
public int read(char cbuf[], int offset, int length) throws IOException
I suggest you start from there
3 Comments
Steakie
This is probably the solution for my problem. got problems implementing tho.. ._. Ill keep trying
Holger
Where’s the relevance of
BufferedInputStream here?
Pavel Niedoba
You right, I should have mentioned FileInputStream, which has buffer on it's own, so theres no there's no need for BufferedInputStream. This might be usefull here: stackoverflow.com/questions/5155226/…
It is not only possible: it is both highly recommended and done in practice. What is usually done is to reuse the same kind of interfaces as Java libraries (InputStream, etc).
In this case, this could mean a new IntegerInputStream class that outputs the digits as a stream. This class can itself forward calls to FileInputStream. Internally, you can use char[] arrays to store the buffer and improve performance, or have calls directed via BufferedInputStream as Pavel suggests, but it is best to isolate the consumer from the internal buffer management and keep the appropriate level of abstraction to the use case (decimals of pi).
Comments
You can open the file with a FileInputStream, and read it byte[] per byte[] to avoid the OOMError.
Comments
According to doc
You should be able to get a String of length Integer.MAX_VALUE (always 2147483647 (231 - 1) by the Java specification, the maximum size of an array, which the String class uses for internal storage) or half your maximum heap size (since each character is two bytes), whichever is smaller
This is why you get the Exception,
if you don't really need the whole 1B chars. you can try using buffer which doesn't load the whole thing into memory.
BufferedReader br = new BufferedReader(new FileReader(new File("path to file")));
char[] data=new char[1000000] ;//however many chars you want;
int i=0;
while ((c = br.read()) != -1 && i<data.length) {
data[i++]= c;
}
br.close();
2 Comments
Holger
Why on earth are you reading up to 1000000 single characters instead of calling just
read(char[]) on the reader? Then, you don’t need a BufferedReader to work around the excessive read() invocations…
nafas
@Holger I guess that's another way of doing it. In terms of complexity both solution should be about the same. but yeah I agree that one liner solution is better
chars, just as you can with an array.