34

Is there an efficient way to remove Nones from numpy arrays and resize the array to its new size?

For example, how would you remove the None from this frame without iterating through it in python. I can easily iterate through it but was working on an api call that would be potentially called many times. 

a = np.array([1,45,23,23,1234,3432,-1232,-34,233,None])
Improve this question
5
  • 2
    What do you want the None's replaced with? Or do you want to remove the None's, then resize the array? Commented Aug 12, 2014 at 1:51
  • I want to remove the Nones and resize the array. Commented Aug 12, 2014 at 1:52
  • 1
    possible duplicate of efficient filter of an array with numpy Commented Aug 12, 2014 at 1:55
  • != is much more efficient than filter Commented Aug 12, 2014 at 2:17
  • Also, you cannot do != None as the previous example has Commented Aug 12, 2014 at 2:18

2 Answers 2

Reset to default
56 
In [17]: a[a != np.array(None)]
Out[17]: array([1, 45, 23, 23, 1234, 3432, -1232, -34, 233], dtype=object)

The above works because a != np.array(None) is a boolean array which maps out non-None values:

In [20]: a != np.array(None)
Out[20]: array([ True,  True,  True,  True,  True,  True,  True,  True,  True, False], dtype=bool)

Selecting elements of an array in this manner is called boolean array indexing.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

If you don't mind me asking, if there are n items in the array, is this method faster than O(n)?
This is saving me 80% of the time on a large array
@wookie919 If I understand the internals of numpy correctly, this is copying out the array, and removing the None's one-by-one. So, I don't think so.
but most of this will live in C. its much faster than filter
Does using a comparison inside of an access (index) operator have a name? Like the colon in the accessor is called the slice.
|
6

I use the following which I find simpler than the accepted answer:

a = a[a != None]

Caveat: PEP8 warns against using the equality operator with singletons such as None. I didn't know about this when I posted this answer. That said, for numpy arrays I find this too Pythonic and pretty to not use. See discussion in comments.

Improve this answer

6 Comments

While it is simpler, this does not work with PEP8 autoformatters like flake8, which would want to turn a[a != None] into a[a is not None] (which is not semantically equivalent).
There is nothing in PEP8 about not using != that I have seen. Consider: sometimes it is apt to use is, other times ==. Similarly, sometimes you need != and other times is not. You just have to know what you are doing. Note also this question/answer pertains to numpy arrays -- for Python lists you need something else.
From PEP8 - "Comparisons to singletons like None should always be done with is or is not, never the equality operators.". I understand that this is talking about numpy, not Python lists.
Your answer is how I used to do this all time... until we started using flake8 (and autopep8) in our repos - this resulted in code changed (sometimes automatically) to a = a[a is not None] which resulted in some weird bugs. Thanks for the link - good discussion!
@DavidSlater I learned something new about PEP8, and added a caveat to my answer. Thanks!
|

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.