Iterating over rows of dataframe but keep each row as a dataframe

Question

I want to iterate over the rows of a dataframe, but keep each row as a dataframe that has the exact same format of the parent dataframe, except with only one row. I know about calling DataFrame() and passing in the index and columns, but for some reason this doesn't always give me the same format of the parent dataframe. Calling to_frame() on the series (i.e. the row) does cast it back to a dataframe, but often transposed or in some way different from the parent dataframe format. Isn't there some easy way to do this and guarantee it will always be the same format for each row?

Here is what I came up with as my best solution so far:

    def transact(self, orders):
    # Buy or Sell
    if len(orders) > 1:
        empty_order = orders.iloc[0:0]
        for index, order in orders.iterrows():
            empty_order.loc[index] = order
            #empty_order.append(order)
            self.sub_transact(empty_order)
    else:
        self.sub_transact(orders)

In essence, I empty the dataframe and then insert the series, from the For loop, back into it. This works correctly, but gives the following warning:

C:\Users\BNielson\Google Drive\My Files\machine-learning\Python-Machine-Learning\ML4T_Ex2_1.py:57: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy empty_order.loc[index] = order C:\Users\BNielson\Anaconda3\envs\PythonMachineLearning\lib\site-packages\pandas\core\indexing.py:477: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy self.obj[item] = s

So it's this line giving the warning:

empty_order.loc[index] = order

This is particularly strange because I am using .loc already, when normally you get this error when you don't use .loc.

Can you provide some same data and code that duplicates your problem? — Scott Boston
– Scott Boston, Commented Apr 23, 2017 at 3:35
Yes, I added the code I came up with that actually works. The truth is this is fairly contextual, so it's a bit hard to explain. The bottom line is that I want each row of a dataframe to have the exact same indexing (though only for one row) and columns. This way my code doesn't have to ask if I have a Series or a Dataframe and them work differently depending on which I have. — Bruce Nielson
– Bruce Nielson, Commented Apr 23, 2017 at 4:17
It's particularly frustrating to get that warning, especially when the documentation suggests you only get it if you don't use .loc (which I am using.) What is it wanting me to use syntactically here? — Bruce Nielson
– Bruce Nielson, Commented Apr 23, 2017 at 4:18

Bruce Nielson · Accepted Answer · 2017-10-07 19:57:39Z

There is a much much easier way to do what I want.

order.to_frame().T

So...

        if len(orders) > 1:
        for index, order in orders.iterrows():
            self.sub_transact(order.to_frame().T) 
    else:
        self.sub_transact(orders)

What this actually does is translates the series (which still contains the necessary column and index information) back to a dataframe. But for some Moronic (but I'm sure Pythonic) reason it transposes it so that the previous row is now the column and the previous columns are now multiple rows! So you just transpose it back.

piRSquared · Accepted Answer · 2017-04-23 05:14:44Z

0

Use groupby with a unique list. groupby does exactly what you are asking for as in, it iterates over each group and each group is a dataframe. So, if you manipulate it such that you groupby a value that is unique for each and every row, you'll get a single row dataframe when you iterate over the group

for n, group in df.groupby(np.arange(len(df))):
    pass
    # do stuff

edited Apr 23, 2017 at 5:14

answered Apr 23, 2017 at 4:55

piRSquared

295k68 gold badges509 silver badges654 bronze badges

Comments

Dharman · Accepted Answer · 2019-09-17 22:14:15Z

0

If I can suggest an alternative way than it would be like this:

for index, order in orders.iterrows():
    orders.loc[index:index]

orders.loc[index:index] is exactly one row dataframe slice with the same structure, including index and column names.

edited Sep 17, 2019 at 22:14

Dharman♦

33.9k27 gold badges103 silver badges155 bronze badges

answered Sep 17, 2019 at 18:38

verymax

11 bronze badge

Collectives™ on Stack Overflow

Iterating over rows of dataframe but keep each row as a dataframe

3 Answers 3

Comments

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Related