Python memory problem

Question

i started programming on python but i have a memory problem (sorry for my bad english). I made a while loop on my algorithm, but on every cicle, the program consummes a lot of memory. I have 3Gb of RAM an AMD 64 x2 processor, and Windows 7 64 bits.

For every cicle, it consummes about 800 Mb of RAM, it's too much i think. Part of my code is here

from sympy import Symbol, diff, flatten 
import numpy as np
from numpy import linalg
from math import log, sqrt, cos, pi
import matplotlib.pyplot as plt

L = 7  #numero de variables
X = [Symbol('x%d' % i) for i in range(L*(L+1))]  #Las variables simbolicas
XX = [X[i] for i in xrange(L)]

LAM = []    

# Parametros
Pr = 10
Eps = 0
Ome = 5
LL = 0.5
b = 2
Gam = 0.2*2*(pi**2)


ran1 = xrange(L)
ran2 = xrange(L*L) 
ran3 = xrange(0,L*(L-1)+1,L)
ran4 = xrange(L,2*L,1)

dt = 0.01
TMAX = 60

def f(x,R,Tau):
    return [Pr*((1 + Eps*cos(Ome*Tau))*x[2] - LL*x[0] - (1 -LL*x[5])) , \
        Pr*((1 + Eps*cos(Ome*Tau))*x[3] - LL*x[1] - (1 - LL)*x[6]),\
        R*x[0] - x[2] - x[0]*x[4],R*x[1] - x[3] - x[1]*x[4],(x[0]*x[2] + x[1]*x[3])/2 - b*x[4],\
        (1/Gam)*(x[0] - x[5]),(1/Gam)*(x[1] - x[6])]

def Jacobian(f,x):          #num son los numeros en el que se evalua la matriz jacobiana, x las variables y f la funcion
    return [[diff(f[i],x[n]) for i in ran1] for n in ran1]

def Y(x):
    return[[x[i+j] for j in ran3] for i in ran4]
    #Ahora la multiplicacion de Y traspuesto por Jacobian traspuesto
def JY(r,Tau):
    J = flatten((np.dot(np.array(Jacobian(f(XX,r,Tau),XX)),np.array(Y(X)))).T)
    return [J[i] for i in ran2]
def Func(x,r,Tau):            #Expandemos las funciones en un solo arreglo
    FFF = []
    map(lambda g: FFF.append(g),f(XX,r,Tau))
    map(lambda g: FFF.append(g),JY(r,Tau))
    return map(lambda f: f.evalf(subs={X[j]:x[j] for j in xrange(L*(L+1))}),FFF)

def RKutta(xi,r):
    i = 1
    while i <= int(TMAX/dt):
        Tau = 0
        YY = xi
        k1 = np.array(Func(YY,r,Tau))*dt
        k2 = (np.array(Func(YY + k1/2,r,Tau/2)))*dt
        k3 = (np.array(Func(YY + k2/2,r,Tau/2)))*dt
        k4 = (np.array(Func(YY + k3,r,Tau)))*dt
        xi = YY + (k1/6) + (k2/3) + (k3/3) + (k4/6)
        Tau = Tau + dt
        i = i + 1
    return [xi[j] for j in xrange(len(xi))]


def lyap(xxi):
    u = [i for i in flatten(np.random.rand(1,L))]
    PhiT = (np.array([[float(xxi[i+j]) for j in ran3] for i in ran4])).T
    PU = np.dot(PhiT,u)
    summ = 0
    jj = 0
    while jj < len(PU):
        summ += (float(PU[jj]))**2
        jj = jj + 1
    lam = log(sqrt(summ))/TMAX
    return lam

R = 46.5
Rmax = 48.5
Rstep = 0.5

while R <= Rmax:
    xi = [5,5,5,5,5,5,5]   #Condiciones Iniciales
    for i in ran2:
        xi.append(None)

    for i in ran4:                    
        for j in ran3:          
            if (i+j+1)%(L+1) == 0:
                xi[i+j] = 1
            else:
                xi[i+j] = 0

    #Ahora el Runge Kutta para integrar todo el sistema


        #Y.append([r for r in xx])
    # savetxt('butterfly.txt', Y, fmt="%12.6G")
    #print Y
    XI = RKutta(xi,R)
    lamb = lyap(XI)
    LAM.append([R,lamb])
    print [R,lamb]
    R = R + Rstep
#print LAM
#x = [LAM[i][0] for i in xrange(len(LAM))]
#y = [LAM[i][1] for i in xrange(len(LAM))]
np.savetxt('lyap3.txt', LAM, fmt="%12.6G")
#plt.axis([10,30,-3,3]);
#plt.scatter(x,y)
#plt.show()

I don't know where the problem could be. Maybe at the Runge Kutta steps or an architecture problem. The memory don't seem to be cleaned at every step and i'm not storing anything, just a pair of numbers at the end of the code. I hope i expressed myself well.

#

OK, i edited this and posted the whole code, i hope someone can help :) . I changed a lot of things, but i still have the memory problem. Each cicle uses about 600 Mb of RAM.

#

Thanks in advance

Answer 1

It's a little tricky to follow the code without context, seeing as you've apparently used numpy in multiple places (both as np and without prefix), and evalf might be from sympy.. but we don't see your imports.

At a very vague guess, some of your list comprehensions build temporary lists that stick around longer than expected. You could perhaps convert those into generators. Another technique is using map() or similar as much as possible.

I also notice a bit of unpythonic index iteration where it's not needed. Func first builds a list called FFF, one item at a time (fairly expensive), then iterates through it by index for no real reason. Use [f(item) for item in seq] rather than [f(seq[i]) for i in xrange(len(seq))], or better yet, map(f, seq), and in this case, try not building the temporary list at all.

Answer 2

What's L? Much of your code uses O(L^2) storage, so if L is large that will be it.

Answer 3

Seems like you are instantiating very large lists. Look if you can't replace some of the list comprehensions with iterators.