Java - How to store huge amounts of values in an array

Question

Hello fellow Stackoverflowers. I have been playing around with arrays in java, and I have been trying to store huge amounts of values in an array. However, I've not been able to store more than a certain amount of values in an array:

String data[] = new String[44681003];//For some reason 44681003 is the highest number I can go to until it spits out an ugly red error message through the console:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

I have a program which generates all the permutations of a given string list, and it works perfectly until I have to generate a number greater than that strange 44680815 number. (For example: 387420489 which is 9^9)

I have tried storing the value and printing it out to the console in a for loop, set the value back to null data[i] = null;

I was just wondering whether there is a way to store larger amounts of values in an array?

OR

Being able to simply print out my value and then remove it from being stored in the array.

Here is my code:

public class Permutations {

public static void main(String[] args) {
    String database = "abcdefghi";

//  String data[] = new String[(int) Math.pow(database.length(), database.length())];
// ^^ I would like to make it this long, but It gives an error.

    String data[] = new String[44681003];
    StringBuilder temp;


    for (int i = 0;i<Math.pow(database.length(), database.length());i++){
        String base = Integer.toString(i,database.length());
        data[i] = base;
        if (base.length()!=database.length()){
             temp = new StringBuilder(""); 
            for (int x = 0;x < (database.length()-data[i].length());x++){
                temp.append('0');
            }
            base = temp + base;

        }

        for (int y = 0;y<database.length();y++){
            base = base.replace((char)('0' + y), database.charAt(y));
        }

        data[i]=null;

        System.out.println("Pos: " + i + "     " + base); //<-- USE THIS TO WRITE IT OUT
    }//end big for loop
    System.out.println("Done");



    }

}

Last lines in console:

Pos: 44680997     badagahcc
Pos: 44680998     badagahcd
Pos: 44680999     badagahce
Pos: 44681000     badagahcf
Pos: 44681001     badagahcg
Pos: 44681002     badagahch
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 44681003
at Permutations.main(Permutations.java:20)

My computer specs: http://store.asus.com/us/item/201510AM160007799/A17602

Windows 10
Intel Core i7 4720HQ 2.6GHz (Turbo up to 3.6GHz)
16GB Memory
1TB HDD
NVIDIA GeForce GTX 970M 3GB
17.3-inch IPS FHD (1920 x 1080) supported G-Sync

Thank you for your time! I hope I can find a solution and maybe help other people with the same/similar question!

Also try to optimize your code. StringBuilder temp; can be declared outside of the loop. — Yassin Hajaj
– Yassin Hajaj, Commented Mar 19, 2016 at 11:20

Peter Lawrey · Accepted Answer · 2016-03-19 12:00:48Z

If you are getting an OutOfMemoryError your program needs more memory to do what you are asking it to do

However, since you don't retain any of the strings you store in the array i.e.

data[i] = null;

I suggest you remove the array as you don't need it. This will solve your memory problem.

You could turn your code into a function so that you don't have to build an array even if you need random access later.

BTW You get N! permutations from a set of N as you cannot repeat any letters. e.g. badagahcc is not a permutation as it has a 3 times and c twice.

public static String generate(String letters, long number) {
    // get a list of the all the possible characters assuming no duplicates.
    List<Character> chars = new ArrayList<>(letters.length());
    for (int i = 0; i < letters.length(); i++)
        chars.add(letters.charAt(i));
    // start with a string builder.
    StringBuilder ret = new StringBuilder(letters.length());

    // while we have characters left
    while(chars.length() > 0) {
       // select one of the unused characters
       int select = number % chars.length();
       // take out the lower portion of the number and use the next portion
       number /= chars.length();
       // Append the N-th character, but remove it so it doesn't get used again.
       ret.append(chars.remove(select));
    }
    assert number == 0; // otherwise we have not enough letters.
    return ret;
}

This way you can get any permutation without memorization.

Thanks for the response! In my program, I would like to generate all possible strings (repetition of chars included). I think i got confused with permutations and combinations. So for me, I would like to generate badagahcc. I will try removing the array and replace it with a string that gets overwritten all the time.
@JoshuaLochner you could use a StringBuilder which gets overwritten however the simplest solution might be to use recursion. Note: it might take a very long time to print them all out given the console isn't very fast.
Could you please elaborate when you mean recursion? (Sorry, I'm not that great with stuff like this xD)... I have successfully removed the array and I am now using 1 string that continually gets overwritten. I am actually busy printing it to the console! It is VERY slow! At 13 million now. (13/44)
@JoshuaLochner updating the console is likely to be your bottleneck. With recursion, you can build up each character in each recursive call.
Oh ok I see! Thank you. And another small question (slightly off topic, but related) Will I be able to write these strings to a file? In other words, can it successfully write 38,7 million to a txt document? I know how to use Buffered reader and other stuff.

Collectives™ on Stack Overflow

Java - How to store huge amounts of values in an array

1 Answer 1

6 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Related