How to avoid Memory Error

Question

How should I proceed in order to be able to plot some data coming out from a big pytable (17GB).

If I try to store the values that I need, I get Memory Error, something like this:

for row in tab.iterrows():

    x.append(row['date'])
    y.append(row['temperature'])

    #this will not end, raises Memory Error

If I try to live update a plot with the data, without storing any values, I get this message:

MemoryError
QImage: out of memory, returning null image

So then? What should I do If I need to plot some data from such a big table?

Note:

Working with python 2.7 32 bits on a Windows 64-bit machine 8GB

I know that a solution would be to use python 64, but, it should be possible to handle in python 32 too.

xtofl · Accepted Answer · 2015-01-27 07:10:20Z

1

Try to find out what section of your data you really want to plot. It doesn't make sense to have > 1 million dots on your screen. You say it yourself: ... if I need to plot __some__ data from such a big table.

I haven't worked with PyTables yet, but the documentation says it's returning numpy objects, so no need to iterate over the rows.

histogram, xedges, yedges = 
   numpy.histogram2d(
      tab.col('date'),
      tab.col('temperature'),
      bins=50)

pyplot.imshow(heatmap)
pyplot.show()

edited Jan 27, 2015 at 7:10

answered Jan 26, 2015 at 13:39

xtofl

41.7k13 gold badges112 silver badges203 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

codeKiller Over a year ago

hey thanks for the answer, but, for this case in particular it makes sense to have many dots in the screen since I am interested in the density...and with some data I wanted to refer to some columns from the pytable, not all columns are of my interest, but thanks again!

xtofl Over a year ago

Then you don't need all data either. You can reduce the data set to NxM counter values, where NxM corresponds to your resolution.

codeKiller Over a year ago

that sounds interesting, but correct me if I am wrong: reducing the data is exactly what I am doing when storing some data with x.append and y.append and as I said, it raises memory error before ending the for loop

xtofl Over a year ago

No; you're copying it. Reducing it can be done with e.g. scipy.stats or numpy.histogram2d or the like.

xtofl Over a year ago

stanford.edu/~mwaskom/software/seaborn/tutorial/…

|

Collectives™ on Stack Overflow

How to avoid Memory Error

1 Answer 1

6 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Related