OutOfMemoryError for creating native threads

Question

We are currently having an issue where we are getting an OutOfMemoryError as follows

java.lang.OutOfMemoryError: unable to create new native thread

This is for a spring integration application

When I took a heap dump I noticed that the following "From Space" was almost always 99% full when checked after the server has just started. All of the following stats are as soon as the server started.

Attaching to process ID 24167, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 20.2-b06

using thread-local object allocation.
Parallel GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 1887436800 (1800.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 536870912 (512.0MB)
   MaxPermSize      = 1073741824 (1024.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 488112128 (465.5MB)
   used     = 294023920 (280.40306091308594MB)
   free     = 194088208 (185.09693908691406MB)
   60.23696260216669% used
From Space:
   capacity = 44695552 (42.625MB)
   used     = 44676144 (42.60649108886719MB)
   free     = 19408 (0.0185089111328125MB)
   99.95657733458577% used
To Space:
   capacity = 70516736 (67.25MB)
   used     = 0 (0.0MB)
   free     = 70516736 (67.25MB)
   0.0% used
PS Old Generation
   capacity = 715849728 (682.6875MB)
   used     = 16172256 (15.423065185546875MB)
   free     = 699677472 (667.2644348144531MB)
   2.2591691199189783% used
PS Perm Generation
   capacity = 536870912 (512.0MB)
   used     = 74156104 (70.72077178955078MB)
   free     = 462714808 (441.2792282104492MB)
   13.812650740146637% used

I am ASSUMING that this is the likeliest cause for the out of memory errors when attempting to create native threads.

Additionally when I checked the memory used by the process via the "top" command it was never 100% and was around 30 to 40% percent.

Any suggestions

Regards, Milinda

Answer 1

This is a very annoying quirk of JVM because the wording "OutOfMemoryError" confuses people (they think there's not enough heap etc.). In reality, this error just means this: "unable to create new native thread".

The reason why it can't can vary, but if you encounter this the correct solution is to find out why there are so many threads and limit the number of them. Even if you manage to create a bit more threads by tuning, your app.s performance will suffer with so many threads.

You can use a debugger to see how many threads are alive. They may have descriptive name, and you can see what code they are executing. You can also find out by which thread they were created. Once you find out where the threads are created, fix it so that it doesn't create so many threads.

Answer 2

Every thread needs stack space, set by -Xss. Strangely, my

java -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal -XX:+LogVMOutput | grep tack

gives 0 for intx ThreadStackSize, but 2048 if I specify -Xss2M. Anyway, there is a default of maybe 64k and all stacks from all threads must fit in your maximum process size which, for 32bit processes in Win32, is limited to 1.5GB.

Answer 3

The question is a bit outdated but if you come here because of a problem like this here is a little hint to check Linux system limits. In Linux there are mainly two sources of limits: 1. ulimit 2. systemd. The first one is well documented in the net, there are process limits and file limits. The second one is new since 2016, systemd defines a DefaultTasksMax limit. This might affect multi threaded java applications and services. In /etc/systemd/system.conf the DefaultTasksMax can be configured system wide. However it is recommend to adjust the TasksMax setting in the systemd service upstart file under which Java is started.