Linked Questions
12 questions linked to/from Psycopg2, Postgresql, Python: Fastest way to bulk-insert
249
votes
21
answers
284k
views
psycopg2: insert multiple rows with one query
I need to insert multiple rows with one query (number of rows is not constant), so I need to execute query like this one:
INSERT INTO t (a, b) VALUES (1, 2), (3, 4), (5, 6);
The only way I know is
...
- 2,989
225
votes
13
answers
346k
views
Bulk insert with SQLAlchemy ORM
Is there any way to get SQLAlchemy to do a bulk insert rather than inserting each individual object. i.e.,
doing:
INSERT INTO `foo` (`bar`) VALUES (1), (2), (3)
rather than:
INSERT INTO `foo` (`bar`...
- 3,819
77
votes
4
answers
109k
views
Parameterized queries with psycopg2 / Python DB-API and PostgreSQL
What's the best way to make psycopg2 pass parameterized queries to PostgreSQL? I don't want to write my own escpaing mechanisms or adapters and the psycopg2 source code and examples are difficult to ...
- 4,256
16
votes
2
answers
14k
views
Efficiently insert massive amount of rows in Psycopg2
I need to efficiently insert about 500k (give or take 100k) rows of data into my PostgreSQL database. After a generous amount of google-ing, I've gotten to this solution, averaging about 150 (wall-...
- 314
14
votes
3
answers
11k
views
psycopg2 - Inserting list of dictionaries into PosgreSQL database. Too many executions?
I am inserting a list of dictionaries to a PostgreSQL database. The list will be growing quickly and the number of dict values (columns) is around 30. The simplified data:
projects = [
{'name': '...
- 2,600
2
votes
4
answers
5k
views
Batch/Bulk SQL insert in Scrapy Pipelines [PostgreSQL]
I am using my own pipeline to store the scrapped items into a PostgreSQL Database, I made an expansion a few days ago and I store the data into a 3 Databases now. So, I want to make the pipeline which ...
- 1,060
5
votes
1
answer
5k
views
Python multi-threading with Postgresql
I have a very basic application that downloads data from a server via a socket connection one record at a time and writes each record to a postgresql database. This is only used by me.
downloader = ...
- 14.7k
3
votes
3
answers
2k
views
Errors inserting many rows into postgreSQL with psycopg2
I have a a number of XML files I need to open and then process to produce a large number of rows that are then inserted into several tables in a remote postgress database.
To extract the XML data I ...
- 610
5
votes
1
answer
4k
views
How to efficiently UPDATE a column in a large PostgreSQL table using Python / psycopg2?
I have a large table with ca. 10 million rows in PostgreSQL 9.4 database. It looks somewhat like this:
gid | number1 | random | result | ...
1 | 2 | NULL | NULL | ...
2 | 15 | ...
- 5,354
6
votes
1
answer
3k
views
Tuning Postgresql performance and memory use in a python workflow
I use Postgresql 9.4 for a model database. My table looks somewhat like this:
CREATE TABLE table1 (
sid INTEGER PRIMARY KEY NOT NULL DEFAULT nextval('table1_sid_seq'::regclass),
col1 INT,
col2 INT,
...
- 5,354
2
votes
1
answer
3k
views
Python psycopg2: Copy result of query to another table
I am having some problem with psycopg2 in python
I have two disparate connections with corresponding cursors:
1. Source connection - source_cursor
2. Destination connection - dest_cursor
Lets say ...
- 863
0
votes
3
answers
133
views
Script only entering last value to postgres database
Very new to python. I am trying to get the data from an RSS feed, parse the data and then insert the data to a database. My short bit of code gets the correct items and I can print the results but I ...
- 57