Reclaiming Memory of Large Objects In Python - Windows and Ubuntu Differences

Question

When I run the following code on Ubuntu and watch in htop, it peaks out at 900 mb of memory usage until sleep() is done:

import random
import gc
import time

large_dict = {}
for x in xrange(50000):
    large_row = {}
    for y in xrange(125):
        large_row[random.randint(1, 10000000000)] = random.randint(1, 10000000000)
    large_dict[random.randint(1, 10000000000)] = large_row


# Force a sweep
large_dict = {}
del large_dict
gc.collect()

time.sleep(60)

However, when I run it on Windows and watch task manager, memory usage is only a few mb after gc.collect().

I tried this code in Ubuntu with Python 2.7.3, 2.7.4, and 2.7.8 with the same results.

Why are Windows and Ubuntu behaving differently? I would prefer Ubuntu to act like Windows and free the memory after gc.collect().

pawl · Accepted Answer · 2014-10-14 02:20:35Z

1

The fact that the free lists grow does not seem like much of a problem because the memory it contains is still accessible to the Python program. But from the OS’s perspective, your program’s size is the total (maximum) memory allocated to Python. Since Python returns memory to the OS on the heap (that allocates other objects than small objects) only on Windows, if you run on Linux, you can only see the total memory used by your program increase.

http://deeplearning.net/software/theano/tutorial/python-memory-management.html

answered Oct 14, 2014 at 2:20

pawl

3544 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Reclaiming Memory of Large Objects In Python - Windows and Ubuntu Differences

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related