python - memory error

Question

When trying to load a large text file into memory I get this:

Python(24297,0xa0d291a8) malloc: *** mach_vm_map(size=717418496) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug 
MemoryError

The code that's causing it is:

with open('list.txt') as f:
        total = sum(1 for _ in f)

Is there a native python way to take care of this?

Answer 1

Wild guess:

You are running the above code on a binary file that contains no (or very few) newlines. Thus, the attempt to read one line reads one very long line.

Try this instead:

with open('list.txt') as f:
    total = sum(block.count('\n')
                for block in iter(lambda: f.read(64*1024), ''))