148 questions
2
votes
2
answers
197
views
Error "table ... already exits" when using to_sql(...,if_exists='append') with Pandas and SQLite
Using Panadas 2.2.3, sqlite3.version 2.6.0 and python 3.12.5,
I get an error "table ... already exits" when using to_sql with if_exists='append'. I just try to append some data from a Pandas ...
- 81
0
votes
2
answers
283
views
How to set a primary key in a database created by pandas to_sql() method?
The to_sql() method for Pandas Dataframe is extremely useful, as shown by the example from the documentation
import pandas as pd
from sqlalchemy import create_engine
# Create sqlite engine
engine = ...
- 129
2
votes
1
answer
889
views
Using pandas.to_sql() and getting TypeError: not all arguments converted during string formatting
I am trying to load a pandas DataFrame to a Snowflake database. I am using sqlalchemy to create the engine. Based in what other StackOverflow answers have suggested I am using with engine.begin() to ...
- 21
1
vote
2
answers
2k
views
Save pandas DataFrame into Postgres running on AirFlow using df.to_sql?
I have a task running on AirFlow that has two steps:
fetches data from MSSql server as a dataframe;
stores it in a PostGres database;
I'm using the MsSqlHook and PostgresHook to manage the ...
2
votes
1
answer
1k
views
Is it possible to create foreign keys with pandas to_sql
I have two DataFrames that have a ForeignKey based relationship, a field in the first DataFrame is a ForeignKey to the Id of the second DataFrame. When I use pandas's to_sql method, The relationship ...
- 3,701
0
votes
1
answer
329
views
column order relevant in df.to_sql(if_exists="append")?
I was wondering, whether df.to_sql("db_table", con=some_engine, if_exists="append") requires that the order of the columns of df (a dataframe) and db_table (a database table) has ...
- 2,117
0
votes
0
answers
150
views
Faster ways to transfer results from a dataframe to SQL Server?
I'm trying to upload 13,000 rows to a SQL Server 2019 (v15.0.5893) using Python.
It takes about three minutes, which seems unreasonably long, and I'm sure it could be done faster. I am using ...
- 1,617
0
votes
1
answer
670
views
SQLALCHEMY Cannot insert explicit value for identity column in table 'tblsummary' when IDENTITY_INSERT is set to OFF
I'm totally new to Pandas and SQL Alchemy.
I'm trying to write code to import Microsoft Access database to SQL Server.
Using the pyodbc and Pandas dataframe approach it's working fine but it's ...
- 83
0
votes
1
answer
287
views
Using pandas to_sql to ignore bad lines on writing to database
I have a pretty huge dataset that needs to be uploaded for database (in my case Postgres). If I use standard to_sql with chunksize of 5000, there will be some bad lines because some data is corrupted. ...
1
vote
1
answer
2k
views
Why is pandas.to_sql trying to create a table even though I specified to append if_exists?
Here's my code. Simply trying to append a dataframe to a Teradata table.
import pandas as pd
from sqlalchemy import create_engine
import sqlalchemy_teradata
user = username
pasw = password
host = ...
- 33
0
votes
0
answers
129
views
Data loaded to Azure SQl database from pandas dataframe writes with Accents in the database
I have trying to load my pandas dataframe to my Azure Sql database using the library BCPANDAS. The problem I'm facing is that the bcpandas writes it to tables with Accents for string columns. I have ...
0
votes
1
answer
967
views
Load dataframe with timestamp and timezone into postgres database with "to_sql"
I have a pandas dataframe containing sensor data with timestamp columns
id time temp humi
min ...
- 143
1
vote
1
answer
738
views
What does a negative return value mean when calling to_sql() in Pandas?
I'm sending various data frames to Microsoft SQL Server using the Pandas function to_sql() and a mssql+pyodbc:// connection made with sqlalchemy.create_engine. Sometimes to_sql() returns the number ...
- 23.3k
0
votes
1
answer
43
views
How do I make sure not only the last element of a for loop dataframe is posted via to_sql to the database?
I try to post the dataframe to pgadmin. The dataframe was edited in a for loop, but when I post the data it posts only the last element in the for loop. I tried to use the to_sql inside the for loop ...
- 75
0
votes
1
answer
74
views
Problem loading a function in Python to_sql
I have the following code in two scripts called Playing_DB_Around.py with the following code:
import data_base.Calc_Func as calc
import pandas as pd
df = pd.DataFrame({'Name': ['John', 'Jane', 'Jim'], ...
- 1,511