read a huge csv and create a dataframe

Question

I have a csv about 4000,0000 rows and 3 columns.I want to read into python,and create a dataframe with these data. I always has memory error.

df = pd.concat([chunk for chunk in pd.read_csv(cmct_0430x.csv',chunksize=1000)])

I also tried creat pandas DataFrame from generator,it still has memory error.

for line in open("cmct_0430x.csv"):
        yield line

my computer is win64,8G

how can I solve this problem? thank you very much.

as answeared, the size should be OK. and if the size is beyond the memory capacity - than it's impossible to store the data frame it self in memory. I would guess there are some issues with the format of the file - for example, reading a file with a non default encoding (ascii or utf8) can cause similar error messages — Ophir Yoktan
– Ophir Yoktan, Commented Nov 16, 2016 at 7:39
Does it work ok with a smaller version of your file? And even if you do manage to load the whole thing into a Dataframe, you might not have enough room to work with the frame. — hpaulj
– hpaulj, Commented Nov 16, 2016 at 7:56
it work with a smaller file.yes,I guess I may do not have enough room to work with it. — lemon
– lemon, Commented Nov 16, 2016 at 8:34
try pd.read_csv('cmct_0430x.csv', nrows=1000000), and some other values for 1000000. See where it crashes — Laurens Koppenol
– Laurens Koppenol, Commented Nov 16, 2016 at 10:24

Laurens Koppenol · Accepted Answer · 2016-11-16 07:32:42Z

1

df = pd.read_csv('cmct_0430x.csv')

40 million rows shouldn't be a problem.

please post your error message if this doesn't work

answered Nov 16, 2016 at 7:32

Laurens Koppenol

3,0912 gold badges23 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

lemon Over a year ago

CParserError: Error tokenizing data. C error: out of memory

amin · Accepted Answer · 2016-11-16 07:43:50Z

1

You actually read the csv file with chunked mode, but merged them into one data-frame in RAM. So the problem still exists. You can divide your data into multiple frames, and work on them separately.

reader = pd.read_csv(file_name, chunksize=chunk_size, iterator=True)

while True:
    try:
        df = reader.get_chunk(chunk_size)
        # work on df
    except:
        break
    del df

answered Nov 16, 2016 at 7:43

amin

1,44114 silver badges25 bronze badges

Collectives™ on Stack Overflow

read a huge csv and create a dataframe

2 Answers 2

1 Comment

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Related