Python: Writing to CSV file from iterating through Pandas Dataframe

Question

I am new to Python. How can I write to a CSV file while iterating through a DataFrame?

readCSV1 = pd.read_csv(r'ProductDetails1.csv')
df1 = DataFrame(readCSV1)

for index, row in df1.iterrows():
print(index, row['SKU'], row['SKU'],row['Title'],row['Cost'],row['Weight 
    Value'],row['Weight Unit']...............)

I am iterating through a large csv file that contains many columns, however I would like to be able to limit the columns in the final csv file as well as put in other arguments. The print output to the console is correct, however I am having trouble figuring out how best to print this to a new csv file as I will be performing many other functions and do not want to simply edit the dataframe itself.

You should not write to the csv file line by line. It is highly inefficient. You should instead clean / limit your dataframe to the contents you want to write, and then call pd.DataFrame.to_csv() all at once: pandas.pydata.org/pandas-docs/stable/generated/… — rahlf23
– rahlf23, Commented Oct 12, 2018 at 18:52
The problem is that I need to iterate through each line as certain variables could affect the item below it. — Tonald Drump
– Tonald Drump, Commented Oct 12, 2018 at 18:54
Then you should perform those operations on the dataframe once you have read it in, then write back once. If you are having issues with specific parts of that logic, then you should update your question to reflect those. — rahlf23
– rahlf23, Commented Oct 12, 2018 at 18:55

rahlf23 · Accepted Answer · 2018-10-12 18:56:36Z

2

You should not write to the csv file line by line. It is highly inefficient. You should instead clean / limit your dataframe to the contents you want to write, and then call pd.DataFrame.to_csv() all at once.

For example:

df = pd.read_csv(r'ProductDetails1.csv')

#Refine your dataframe here
refined_df = df[['SKU','Title','Close','Weight Value','Weight Unit']]

refined_df.to_csv(r'ProductDetails1.csv', index=False)

answered Oct 12, 2018 at 18:56

rahlf23

9,0394 gold badges30 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

jpp · Accepted Answer · 2018-10-12 18:55:31Z

1

I would like to be able to limit the columns in the final csv file as well as put in other arguments.

Don't make these adjustments as you write your CSV file. Make the adjustments in the dataframe itself before you export to CSV. Then use the purpose-built pd.DataFrame.to_csv method.

For example, you can select specific columns in a specific order before exporting:

df = df[['SKU', 'SKU', 'Title', 'Cost']]
df.to_csv('file.csv', index=False)

To "put in other arguments", make new columns via appropriate logic.

answered Oct 12, 2018 at 18:55

jpp

166k37 gold badges301 silver badges362 bronze badges

2 Comments

Tonald Drump Over a year ago

I would also need to implement conditional logic to determine the contents of one of my columns, how best do I implement this in a data frame?

jpp Over a year ago

@TonaldDrump, I really can't answer on such limited information. I suggest you read a Pandas tutorial; and, if you're still stuck and can't find an answer on SO, ask a new question.

Collectives™ on Stack Overflow

Python: Writing to CSV file from iterating through Pandas Dataframe

2 Answers 2

Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Related