Boolean indexing on multiple pandas columns [duplicate]

Question

despite there being at least two good tutorials on how to index a DataFrame in Python's pandas library, I still can't work out an elegant way of SELECTing on more than one column.

>>> d = pd.DataFrame({'x':[1, 2, 3, 4, 5], 'y':[4, 5, 6, 7, 8]})
>>> d
   x  y
0  1  4
1  2  5
2  3  6
3  4  7
4  5  8
>>> d[d['x']>2] # This works fine
   x  y
2  3  6
3  4  7
4  5  8
>>> d[d['x']>2 & d['y']>7] # I had expected this to work, but it doesn't
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I have found (what I think is) a rather inelegant way of doing it, like this

>>> d[d['x']>2][d['y']>7]

But it's not pretty, and it scores fairly low for readability (I think).

Is there a better, more Python-tastic way?

Asclepius · Accepted Answer · 2017-06-15 20:41:52Z

93

It is a precedence operator issue.

You should add extra parenthesis to make your multi condition test working:

d[(d['x']>2) & (d['y']>7)]

This section of the tutorial you mentioned shows an example with several boolean conditions and the parenthesis are used.

edited Jun 15, 2017 at 20:41

Asclepius

64.5k20 gold badges188 silver badges164 bronze badges

answered Jun 20, 2013 at 14:43

Zeugma

32.3k9 gold badges73 silver badges85 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Harry Moreno Over a year ago

note: the unary operator & is required here. d[(d['x']>2) and (d['y']>7)] fails with ValueError: The truth value of a Series is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

TomAugspurger · Accepted Answer · 2013-06-20 14:33:44Z

2

There may still be a better way, but

In [56]: d[d['x'] > 2] and d[d['y'] > 7]
Out[56]: 
   x  y
4  5  8

works.

answered Jun 20, 2013 at 14:33

TomAugspurger

29k8 gold badges89 silver badges71 bronze badges

3 Comments

Jeff Over a year ago

this works, but ends up using python operators (rather than numpy) and so is going to be much slower

LondonRob Over a year ago

that's a nice solution. I like the fact that it explicitly uses and. Makes it clearer that there are two conditions being evaluated.

LondonRob Over a year ago

Oh, I've just found a duplicate of this question. Whoops.

Collectives™ on Stack Overflow

Boolean indexing on multiple pandas columns [duplicate]

2 Answers 2

1 Comment

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Linked

Related