How import multiple text files to one dataframe in python?

Question

I found here how to import multiple text files to one data frame. However, it gives an error. Files are with the names as footballseason1,footballseason2,footballseason3 ... (until footballseason5000)

import pandas as pd
import datetime as dt
import os, glob
os.chdir("~/Downloads/data")
filenames = [i for i in glob.glob("*.txt")]

FileNotFoundError: [Errno 2] No such file or directory: '~/Downloads/data'

However, if I try to import one file, everything is working and the directory is found

df = pd.read_csv("~/Downloads/data/footballseason1.txt", sep=",")

Could you help to fix the problem? and are there any ways to do it without changing directory and simply do all the steps using the path where all files are located?

Answer 1

Python's os does not understand ~ by default, so it needs to be expanded manually:

filenames = [i for i in glob.glob(os.path.expanduser("~/Downloads/data/*.txt"))]

Answer 2

You can use python's list comprehension and pd.concat like below

df = pd.concat([pd.read_csv(i, sep=',') for i in glob.glob("~/Downloads/data/*.txt", recursive=True)])

Answer 3

Via pathlib ->

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
df = pd.concat([
    pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
])

With added check - >

import pandas as pd
from pathlib import Path
inp_path = Path("~/Downloads/data")
if inp_path.exists():
    df = pd.concat([
        pd.read_csv(txt_file, sep=',') for txt_file in inp_path.glob('*.txt')
    ])
else:
    print('input dir doesn\'t exist please check path')

Answer 4

Importing Data from Multiple files Now let’s see how can we import data from multiple files from a specific directory. There are many ways to do so, but I personally believe this is an easier and simpler way to use and also to understand especially for beginners. 1)First, we are going to import the OS and glob libraries. We need them to navigate through different working directories and getting their paths. import os import glob 2) We also need to import the pandas library as we need to work with data frames. import pandas as pd 3) Let’s change our working directory to the directory where we have all the data files. os.chdir(r"C:\Users\HARISH\Path_for_our_files") 4) Now we need to create a for loop which iterates through all the .csv file in the current working directory filenames = [i for i in glob.glob("*.csv")]