Pandas append() error with two dataframes

Question

When I try to append two or more dataframe and output the result to a csv, it shows like a waterfall format.

    dataset = pd.read_csv('testdata.csv')
  for i in segment_dist:
      for j in step:
          print_msg = str(i) + ":" + str(j)
          print("\n",i,":",j,"\n")
          temp = pd.DataFrame(estimateRsq(dataset,j,i),columns=[print_msg])
          csv = csv.append(temp)
  csv.to_csv('output.csv',encoding='utf-8', index=False)

estimateRsq() returns array. I think this much code snippet should be enough to help me out.

The format I am getting in output.csv is:

Please help, How can I shift the contents go up from index 1.

koPytok · Accepted Answer · 2018-05-24 07:48:04Z

1

From df.append documentation:

Append rows of other to the end of this frame, returning a new object. Columns not in this frame are added as new columns.

If you want to add column to the right, use pd.concat with axis=1 (means horizontally):

list_of_dfs = [first_df, second_df, ...]
pd.concat(list_of_dfs, axis=1)

You may want to add parameter ignore_index=True if indexes in dataframes don't match.

edited May 24, 2018 at 7:48

answered May 24, 2018 at 7:36

koPytok

3,7531 gold badge16 silver badges29 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jpp · Accepted Answer · 2018-05-24 10:06:33Z

1

Build a list of dataframes, then concatenate

pd.DataFrame.append is expensive relative to list.append + a single call of pd.concat.

Therefore, you should aggregate to a list of dataframes and then use pd.concat on this list:

lst = []

for i in segment_dist:
    # do something
    temp = pd.DataFrame(...)
    lst.append(temp)

df = pd.concat(lst, ignore_index=True, axis=0)

df.to_csv(...)

answered May 24, 2018 at 10:06

jpp

166k37 gold badges301 silver badges362 bronze badges

Collectives™ on Stack Overflow

Pandas append() error with two dataframes

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related