Read multiple csv files from multiple folders in Python [duplicate]

Question

I have a folder that includes folders and these folders include many csv files. I want to import and concatenate all of them in Python.

Let's say main folder: /main

subfolders: /main/main_1

csv: /main/main_1/first.csv

path='/main'
df_list = [] 
for file in os.listdir(path): 
    df = pd.read_csv(file) 
    df_list.append(df) 

final_df = df.append(df for df in df_list) 

Answer 1

What about this:

import pandas as pd
from pathlib import Path

directory = "path/to/root_dir"

# Read each CSV file in dir "path/to/root_dir"
dfs = []
for file in Path(directory).glob("**/*.csv"):
    dfs.append(pd.read_csv(file))

# Put the dataframes to a single dataframe
df = pd.concat(dfs)

Change the path/to/root_dir to where ever your CSV files are.

Answer 2

I found a way to concat all of them but it doesn't satisfy to me as it takes too much time due to computational complexity.

path = "/main"

folders = []
directory = os.path.join(path)
for root,dirs,files in os.walk(directory):
    folders.append(root)

del folders[0]


final = []
for folder in folders:
    df = pd.concat(map(pd.read_csv, glob.glob(os.path.join(folder + "/*.csv"))))
    final.append(df)

Answer 3

Remember to add back main to the path:

df =pd.read_csv(path + "/" + file)