Skip to main content
edited tags
Link
jarlh
  • 44.9k
  • 8
  • 52
  • 68
Source Link

Pandas sql update table efficiently

I am using python pandas and pyodbc to load data from a SQL Server database and bulk the data to a csv, then, I need to update a table using the values contained in a list. There are a 10,000+ rows so the UPDATE query's take some time.

Is there an efficient way to make an update statement for each row value from a column contained in the df?

I have created a list containing the values I need to update.

This is the statement that I want to execute:

 UPDATE MYTABLE SET ResRet ='1' WHERE ResTip ='FC' AND ResNum = --'#column_values'

Here´s my code:

import pyodbc
import pandas as pd
pd.set_option('display.max_rows',None) 

server = #myserver 
database = #mydb 
username = #myuser 
password = #mypwd 
conn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';DATABASE='+database+';UID='+username+';PWD='+ password)

docs_pd = pd.read_csv("data/processed/"+"cufe_vacio.csv", usecols = [4,5,16,17,30])
docs_pd["ResNum"] = docs_pd['ResNum'].astype('int')
docs_pd["ResFec"] = docs_pd['ResFec'].astype('datetime64')
docs_pd["RESFECMOV"] = docs_pd['RESFECMOV'].astype('datetime64')

filter_df = docs_pd.apply(lambda x: x.str.strip() if x.dtype == "object" else x)  # Trim whitespaces

values = filter_df['ResNum'].value_counts()
labels = values.index.unique()