Matplotlib runs out of memory when plotting in a loop

Question

I have a fairly simple plotting routine that looks like this:

from __future__ import division
import datetime
import matplotlib
matplotlib.use('Agg')
from matplotlib.pyplot import figure, plot, show, legend, close, savefig, rcParams
import numpy
from globalconstants import *

    def plotColumns(columnNumbers, t, out, showFig=False, filenamePrefix=None, saveFig=True, saveThumb=True):
        lineProps = ['b', 'r', 'g', 'c', 'm', 'y', 'k', 'b--', 'r--', 'g--', 'c--', 'm--', 'y--', 'k--', 'g--', 'b.-', 'r.-', 'g.-', 'c.-', 'm.-', 'y.-', 'k.-']

        rcParams['figure.figsize'] = (13,11)
        for i in columnNumbers:
            plot(t, out[:,i], lineProps[i])

        legendStrings = list(numpy.zeros(NUMCOMPONENTS)) 
        legendStrings[GLUCOSE] = 'GLUCOSE'
        legendStrings[CELLULOSE] = 'CELLULOSE'
        legendStrings[STARCH] = 'STARCH'
        legendStrings[ACETATE] = 'ACETATE'
        legendStrings[BUTYRATE] = 'BUTYRATE'
        legendStrings[SUCCINATE] = 'SUCCINATE'
        legendStrings[HYDROGEN] = 'HYDROGEN'
        legendStrings[PROPIONATE] = 'PROPIONATE'
        legendStrings[METHANE] = "METHANE"

        legendStrings[RUMINOCOCCUS] = 'RUMINOCOCCUS'
        legendStrings[METHANOBACTERIUM] = "METHANOBACTERIUM"
        legendStrings[BACTEROIDES] = 'BACTEROIDES'
        legendStrings[SELENOMONAS] = 'SELENOMONAS'
        legendStrings[CLOSTRIDIUM] = 'CLOSTRIDIUM'

        legendStrings = [legendStrings[i] for i in columnNumbers]
        legend(legendStrings, loc='best')

        dt = datetime.datetime.now()
        dtAsString = dt.strftime('%d-%m-%Y_%H-%M-%S')

        if filenamePrefix is None:
            filenamePrefix = ''

        if filenamePrefix != '' and filenamePrefix[-1] != '_':
            filenamePrefix += '_'

        if saveFig: 
            savefig(filenamePrefix+dtAsString+'.eps')

        if saveThumb:
            savefig(filenamePrefix+dtAsString+'.png', dpi=300)


        if showFig: f.show()

        close('all')

When I plot this in single iterations, it works fine. However, the moment I put it in a loop, matplotlib throws a hissy fit...

Traceback (most recent call last):
  File "c4hm_param_variation_h2_conc.py", line 148, in <module>
    plotColumns(columnNumbers, timeVector, out, showFig=False, filenamePrefix='c
4hm_param_variation_h2_conc_'+str(hydrogen_conc), saveFig=False, saveThumb=True)

  File "D:\phdproject\alexander paper\python\v3\plotcolumns.py", line 48, in plo
tColumns
    savefig(filenamePrefix+dtAsString+'.png', dpi=300)
  File "C:\Python25\lib\site-packages\matplotlib\pyplot.py", line 356, in savefi
g
    return fig.savefig(*args, **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 1032, in savef
ig
    self.canvas.print_figure(*args, **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\backend_bases.py", line 1476, i
n print_figure
    **kwargs)
  File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
358, in print_png
    FigureCanvasAgg.draw(self)
  File "C:\Python25\lib\site-packages\matplotlib\backends\backend_agg.py", line
314, in draw
    self.figure.draw(self.renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\figure.py", line 773, in draw
    for a in self.axes: a.draw(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\axes.py", line 1735, in draw
    a.draw(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\artist.py", line 46, in draw_wr
apper
    draw(artist, renderer, *kl)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 374, in draw
    bbox = self._legend_box.get_window_extent(renderer)
  File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 209, in get
_window_extent
    px, py = self.get_offset(w, h, xd, yd)
  File "C:\Python25\lib\site-packages\matplotlib\offsetbox.py", line 162, in get
_offset
    return self._offset(width, height, xdescent, ydescent)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 360, in findof
fset
    return _findoffset(width, height, xdescent, ydescent, renderer)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 325, in _findo
ffset_best
    ox, oy = self._find_best_position(width, height, renderer)
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 817, in _find_
best_position
    verts, bboxes, lines = self._auto_legend_data()
  File "C:\Python25\lib\site-packages\matplotlib\legend.py", line 669, in _auto_
legend_data
    tpath = trans.transform_path(path)
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1911, in t
ransform_path
    self._a.transform_path(path))
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1122, in t
ransform_path
    return Path(self.transform(path.vertices), path.codes,
  File "C:\Python25\lib\site-packages\matplotlib\transforms.py", line 1402, in t
ransform
    return affine_transform(points, mtx)
MemoryError: Could not allocate memory for path

This happens on iteration 2 (counting from 1), if that makes a difference. The code is running on Windows XP 32-bit with python 2.5 and matplotlib 0.99.1, numpy 1.3.0 and scipy 0.7.1.

EDIT: The code has now been updated to reflect the fact that the crash actually occurs at the call to legend(). Commenting that call out solves the problem, though obviously, I would still like to be able to put a legend on my graphs...

Does this answer your question? How can I release memory after creating matplotlib figures — user14093575
– user14093575, Commented Nov 22, 2020 at 13:50

Marioanzas · Accepted Answer · 2023-02-08 12:37:40Z

38

Is each loop supposed to generate a new figure? I don't see you closing it or creating a new figure instance from loop to loop.

This call will clear the current figure after you save it at the end of the loop:

pyplot.clf()

I'd refactor, though, and make your code more OO and create a new figure instance on each loop:

from matplotlib import pyplot

while True:
  fig = pyplot.figure()
  ax = fig.add_subplot(111)
  ax.plot(x,y)
  ax.legend(legendStrings, loc = 'best')
  fig.savefig('himom.png')
  # etc....

edited Feb 8, 2023 at 12:37

Marioanzas

1,9452 gold badges21 silver badges38 bronze badges

answered Mar 2, 2010 at 18:48

Mark

109k20 gold badges180 silver badges238 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Chinmay Kanchi Over a year ago

Bah! I'd forgotten the clf() call... Adding that fixed it.

drevicko Over a year ago

From @geographica's answer it seems that clf() clears the figure, but doesn't release all references to it, unlike close(fig). It won't hold much memory when cleared, but if there's many, it'll add up...

praxmon Over a year ago

So if I have a real time plot of about 30k values and I have other arcs,ellipses.circles that I need to plot, what will pyplot.clf() do exactly? Also for animation I have used pyplot.pause(time_duration). How do I preserve the values I have plotted before and then plot the values again and again as I get the data?

Robin Over a year ago

For me it also helped to manually run Python's garbage collection from time to time to get rid of out of memory errors. For this I added import gc at the top of my code and then gc.collect() after close() (or every now and then, depending on the code).

Hui Liu Over a year ago

I was able to fix my problem with both fig.clf() AND plt.close(fig). Otherwise, by only using plt.close(fig), I still OOMed with more than 10 Gb when plotting and saving ~8000 images in Kaggle notebook.

|

ninjasmith · Accepted Answer · 2011-10-25 12:21:21Z

34

I've also run into this error. what seems to have fixed it is

while True:
    fig = pyplot.figure()
    ax = fig.add_subplot(111)
    ax.plot(x,y)
    ax.legend(legendStrings, loc = 'best')
    fig.savefig('himom.png')
    #new bit here
    pylab.close(fig) #where f is the figure

running my loop stably now with fluctuating memory but no consistant increase

answered Oct 25, 2011 at 12:21

ninjasmith

1,7841 gold badge13 silver badges10 bronze badges

2 Comments

geographika Over a year ago

+1 pyplot.close() released the memory in a loop I had and prevented Python from crashing.

ecoe Over a year ago

even with low memory plots this is necessary (e.g. after generating several hundred). pyplot.close('all') is another approach to consider when plotting multiple figures in a loop

Cristian Ciupitu · Accepted Answer · 2015-10-26 09:37:46Z

Answer from ninjasmith worked for me too - pyplot.close() enabled my loops to work.

From the pyplot tutorial, Working with multiple figures and axes:

You can clear the current figure with clf() and the current axes with cla(). If you find this statefulness, annoying, don’t despair, this is just a thin stateful wrapper around an object oriented API, which you can use instead (see Artist tutorial)

If you are making a long sequence of figures, you need to be aware of one more thing: the memory required for a figure is not completely released until the figure is explicitly closed with close(). Deleting all references to the figure, and/or using the window manager to kill the window in which the figure appears on the screen, is not enough, because pyplot maintains internal references until close() is called.

Innuendo · Accepted Answer · 2023-01-30 11:36:06Z

5

I had a similar issue when I was using it from jupyter, putting plt.clf() and plt.close() in the loop did not work.

But this helped:

import matplotlib
matplotlib.use('Agg')

This disables interactive backend for matplotlib.

answered Jan 30, 2023 at 11:36

Innuendo

6816 silver badges11 bronze badges

2 Comments

Noé AC Over a year ago

Thanks! This solution solved my problem but the other answers above did not. This should be upvoted.

masaya Over a year ago

I also confirmed this solved my problem with matplotlib version '3.7.2'. Thank you for sharing your solution. I think this is the best answer for this question as of 2023!!!

masaya · Accepted Answer · 2023-09-11 05:52:41Z

3

In my case, matplotlib version 3.5.0, As Hui Liu san says,
Following method can keep memory usage low

import matplotlib
print(matplotlib.__version__) #'3.5.0'
import matplotlib.pyplot as plt

plt.savefig('your.png')

# Add both in this order for keeping memory usage low
plt.clf()   
plt.close()

--- Added ----
With matplotlib version '3.7.2', Memory continued to increase even with the above method.
By adding the following code as described on this site(http://datasideoflife.com/?p=1443) and by Innuendo, the memory increase has been eliminated.

import matplotlib
matplotlib.use('Agg') # for avoiding memory leak

edited Sep 11, 2023 at 5:52

answered Jul 6, 2022 at 23:56

masaya

4902 gold badges11 silver badges19 bronze badges

3 Comments

Frank_Coumans Over a year ago

Plt.close() closes the figure and thus also clears the content. So I don't understand why plt.clf() is need here.

DrBwts Over a year ago

@Frank_Coumans I suspect its because even though the window reference is being removed/garbage collected, the reference to the memory that the figure occupies is not hence the need for plt.clf()

masaya Over a year ago

Perhaps there was a bug that did not work as specified. I described the results of my experiments. Recently I found that it did not work in other versions and found other solutions, so I have added them to this page. Thanks guys for your comments.

IljaBek · Accepted Answer · 2024-10-02 15:07:23Z

0

I've tried to combine several approaches on SO to decouple the figures from pyplot in jupyter:

import matplotlib.pyplot as plt
from matplotlib import figure
from IPython.display import display


# loop:
    plot_mos = [ ['one']*5,range(5) ]

    fig = figure.Figure()
    ax = fig.subplot_mosaic(plot_mos)
    
    ### Plotting happens here

    display(fig)

    fig.clear()
    ax.clear()
    plt.close(fig)
    del fig

answered Oct 2, 2024 at 15:07

IljaBek

6211 gold badge11 silver badges21 bronze badges

Collectives™ on Stack Overflow

Matplotlib runs out of memory when plotting in a loop

6 Answers 6

7 Comments

2 Comments

Comments

2 Comments

3 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

7 Comments

2 Comments

Comments

2 Comments

3 Comments

Comments

Linked

Related