11

Using Python3.5.3 in Spyder 3.1.3 on Raspbian on a Raspberry Pi. Appending two numpy-arrays to a list named 'list0' works well with allocated numpy arrays 'a' like:

import numpy as np

list0 = []
a = np.array([[1,2,3],[2,3,4]])
list0.append(a)
a = np.array([[11,12,13],[12,13,14]])
list0.append(a)

print("list0 =",list0)

works well, giving as output (a bit better formatted for the post):

list0 = [ array([[ 1,  2,  3], [ 2,  3,  4]]), 
          array([[11, 12, 13], [12, 13, 14]]) ]

Replacing the assignment to a using a loop, weird things happen:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a)
print("list0 =",list0)

The second line tells Python the shape of the array used (in my original case it is a three-dimensional array). For verification the generated arrays named 'a' are printed out. Appending the newly filled arrays 'a' to 'list0' finally shows four times the last line.

idx = 0 ; a = [ 0  1  2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [ array([30, 31, 32]), array([30, 31, 32]), 
          array([30, 31, 32]), array([30, 31, 32]) ]

I suppose that 'list0' simply contains four times a pointer to the array 'a' which only exists in one instance / memory range.

So: How can I physically append (copy?) each of the different arrays 'a' to the list? Is it a python bug or is it simply my misunderstanding of something? Certainly I should think more pythonian ;c)

Thanks for help, Peter

Improve this question
5
  • 1
    This is a common problem when adding a mutable object to a list. The list stores pointers to the object, which in this case is the same in each loop. a needs to be a new array each iteration. You can save an iteration with a = idx*10 + np.arange(3). Commented Dec 17, 2018 at 23:57
  • 2
    This is a fundamental misunderstanding of Python semantics. Read the following: nedbatchelder.com/text/names.html In one case, you are creating two different arrays, and appending them to your list, In the other, you create a single array, and append it twice two your list, so your list contains two references to the same array. You can copy an array using the .copy method. Commented Dec 17, 2018 at 23:58
  • 1
    @hpaulj I think it's best to not even consider things like pointers in Python (which do not exist in Python) and rather, learn the language on it's own terms. Yes, it is true, the CPython implementation uses some primitive array of PyObject pointers, but tahtis an implementation detail Commented Dec 17, 2018 at 23:59
  • 3
    @juanpa.arrivillaga, in the context of numpy object dtype arrays I think pointer, or reference, is the best terminology. As with a ndarray, the array's databuffer stores itemsize elements regardless of what they reference (or point to). That terminology carries over to my discussion of lists. Commented Dec 18, 2018 at 0:07
  • 2
    That is all true, but a list is not a list of pointers. Pointers are really not germane to python at all. Commented Dec 18, 2018 at 0:09

2 Answers 2

Reset to default
14

The problem

You're appending the same array a to your list0 4 times. Arrays like a are mutable objects, which means, among other things, that when you assign values to them the underlying object changes. Since the array is present in your list 4 times, those changes (seem to) show up in 4 different places.

Solution

You can fix the code you have with one small change. Append a copy of the array to your list, instead of the array itself:

import numpy as np
a = np.empty((3), int)
list0 = []
for idx in range(4):    
    for i in range(3):
        a[i] = idx*10 + i
    print("idx =",idx,"; a =",a)
    list0.append(a.copy())
print("list0 =",list0)

Output:

idx = 0 ; a = [0 1 2]
idx = 1 ; a = [10 11 12]
idx = 2 ; a = [20 21 22]
idx = 3 ; a = [30 31 32]
list0 = [array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

Optimized solution

Python/Numpy offer many better ways (both in terms of using fewer lines of code and running faster) to initialize arrays. For a bunch of ranges like this, here is a reasonable approach:

list0 = [np.arange(n*10, n*10+3) for n in range(4)]
print(list0)

Output:

[array([0, 1, 2]), array([10, 11, 12]), array([20, 21, 22]), array([30, 31, 32])]

You might also consider just using a single 2D array in place of a list of arrays. One single array is often easier to work with than a heterogenous mix of arrays in a list. Here's how you do that:

arr0 = np.array([np.arange(n*10, n*10+3) for n in range(4)])
print(arr0)

Output:

[[ 0  1  2]
 [10 11 12]
 [20 21 22]
 [30 31 32]]
Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Many thanks tel (and others). Helped much in understanding and putting pointers out of my pymind. The .copy() worked fine at once. My data is not generated as in the example but received in a binary array from pyserial and is distributed to a 3D matrix. These are are stacked together for later evaluation. Looking forward to adapt it to a more numpy like and faster method.
7

Just do this:

list_to_append.append(np_array.copy())

In a nutshell, numpy arrays or lists are mutable objects, which means that you when you assign a numpy array or list to a variable, what you are really assigning are references to memory locations aka pointers.

In your case, "a" is a pointer, so what you are really doing is appending to list0 an address to the memory location pointed by "a", and not the actual values pointed by the pointer. Thus it means that each new position of "list0", after appending, turns out to be the same address of memory: "a".

So, instead of:

list0.append(a)

You call the copy() method of "a" that creates a new memory location for the new values of "a" and returns it:

list0.append(a.copy())
Improve this answer

4 Comments

It would be good if you can explain why the OP's approach didn't behave as they expect, as this will also help future readers with this problem. You can then explain why your approach doesn't encounter such issues.
While this code may solve the question, including an explanation of how and why this solves the problem would really help to improve the quality of your post, and probably result in more up-votes. Remember that you are answering the question for readers in the future, not just the person asking now. Please edit your answer to add explanations and give an indication of what limitations and assumptions apply.
Regarding requested explanations on obove answer: your suggestions are helpful, also for me, not to forget some explanation if I can answer a question. Good explanations were posted above; valuable thoughts for my deeper understanding and further reading. But Fsn9 s answer would have helped me in a second, arguing that pointers are not the way to think in Python ;c)
Thanks for the advice. I guess that it's better now.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.