Return to Answer

Revision III:

If string concatenation in StringBuilders is taking inordinately long, perhaps your memory is very full. So our goal is to achieve string concatenation without chewing up a lot of memory. Hopefully the savings in CPU time will follow automatically.

My plan went like this: Instead of concatenating those substrings into a long StringBuilder, you could build a List of references to those (pre-existing) Strings. The list of references should be shorter than the sum of the substrings and thus consume less memory.

Only when it becomes time to store that big String do we concatenate the parts in one big StringBuilder, pull out the String, store the String, throw away the reference to the String, clear the StringBuilder, repeat. I felt this was a brilliant solution!

However, from this article from 2002, a String reference in an array, probably likewise in an ArrayList, takes a whopping 8 bytes! A recent StackOverflow post confirmed that this is still so. Thus, a list of references to 10-byte Strings saves only 2 bytes per String. Thus, I present my "solution" as a possibility for similar problems, but I don't see this particular problem being able to benefit from it.

Revision III:

If string concatenation in StringBuilders is taking inordinately long, perhaps your memory is very full. So our goal is to achieve string concatenation without chewing up a lot of memory. Hopefully the savings in CPU time will follow automatically.

My plan went like this: Instead of concatenating those substrings into a long StringBuilder, you could build a List of references to those (pre-existing) Strings. The list of references should be shorter than the sum of the substrings and thus consume less memory.

Only when it becomes time to store that big String do we concatenate the parts in one big StringBuilder, pull out the String, store the String, throw away the reference to the String, clear the StringBuilder, repeat. I felt this was a brilliant solution!

However, from this article from 2002, a String reference in an array, probably likewise in an ArrayList, takes a whopping 8 bytes! A recent StackOverflow post confirmed that this is still so. Thus, a list of references to 10-byte Strings saves only 2 bytes per String. Thus, I present my "solution" as a possibility for similar problems, but I don't see this particular problem being able to benefit from it.

Another idea:

If memory is limited and 2/3 of the strings are already there and most of those strings are significantly longer than about 16 bytes, then I'd recommend the following:

Represent the combined strings with a linked list of references to the original strings. Only when it comes time to write the combined string to memory, do the concatenation, store it and then throw it away. This way, there will only ever be one "long" string in existence, briefly.

Re-reading the initial question, it looks like the average substring is 10 bytes, so LinkedList won't be suitable. Even hand-rolling your own linked list will only bring the overhead down to about 8 bytes, so this will probably not help.

Revision III:

If string concatenation in StringBuilders is taking inordinately long, perhaps your memory is very full. So our goal is to achieve string concatenation without chewing up a lot of memory. Hopefully the savings in CPU time will follow automatically.

My plan went like this: Instead of concatenating those substrings into a long StringBuilder, you could build a List of references to those (pre-existing) Strings. The list of references should be shorter than the sum of the substrings and thus consume less memory. 

Only when it becomes time to store that big String do we concatenate the parts in one big StringBuilder, pull out the String, store the String, throw away the reference to the String, clear the StringBuilder, repeat. I felt this was a brilliant solution!

However, from this article from 2002, a String reference in an array, probably likewise in an ArrayList, takes a whopping 8 bytes! A recent StackOverflow post confirmed that this is still so. Thus, a list of references to 10-byte Strings saves only 2 bytes per String. Thus, I present my "solution" as a possibility for similar problems, but I don't see this particular problem being able to benefit from it.

Another idea:

If memory is limited and 2/3 of the strings are already there and most of those strings are significantly longer than about 16 bytes, then I'd recommend the following:

Represent the combined strings with a linked list of references to the original strings. Only when it comes time to write the combined string to memory, do the concatenation, store it and then throw it away. This way, there will only ever be one "long" string in existence, briefly.

Another idea:

If memory is limited and 2/3 of the strings are already there and most of those strings are significantly longer than about 16 bytes, then I'd recommend the following:

Represent the combined strings with a linked list of references to the original strings. Only when it comes time to write the combined string to memory, do the concatenation, store it and then throw it away. This way, there will only ever be one "long" string in existence, briefly.

Re-reading the initial question, it looks like the average substring is 10 bytes, so LinkedList won't be suitable. Even hand-rolling your own linked list will only bring the overhead down to about 8 bytes, so this will probably not help.