Load a large DataFrame from Pandas to Postgresql

Question

I have been trying to load a huge DF into postgres table with SqlAlchemy but the process is always been killed after a while. Is there a better way to do this with pandas?

...

>>engine = create_engine('postgresql://stargateuser:5tar9ate@localhost/stargate',encoding='utf-8', echo=True)

>>MainName.to_sql("landingpage_mainname", con=engine, if_exists="replace")
Killed

Have you tried the "chunksize" keyword? i.e. MainName.to_sql("landingpage_mainname", con=engine, if_exists="replace", chunksize=100) Here's the doc for to_sql — John Ching
– John Ching, Commented Aug 22, 2018 at 12:45

NobbyNobbs · Accepted Answer · 2018-08-22 14:07:56Z

3

This works! Thanks @JohnChing

MainName.to_sql("landingpage_mainname", con=engine,
                if_exists="replace", chunksize=200000)

edited Aug 22, 2018 at 14:07

NobbyNobbs

1,4441 gold badge12 silver badges17 bronze badges

answered Aug 22, 2018 at 13:54

road

4793 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Load a large DataFrame from Pandas to Postgresql

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related