Write to CSV from sqlite3 database in python

Question

Ok, So I have a database called cars.db which has a table == inventory,

Inventory essentially contains

    ('Ford', 'Hiluz', 2),
    ('Ford', 'Tek', 6),
    ('Ford', 'Outlander', 9),
    ('Honda', 'Dualis', 3),
    ('Honday', 'Elantre', 4)

I then wrote this which is meant to edit that to the csv, however, I can't seem to work this out, in some cases I get stuff to print but its not right, and when I try and fix that, nothing prints. Any suggestions to get me on track?

#write table to csv

import sqlite3
import csv

with sqlite3.connect("cars.db") as connection:
    csvWriter = csv.writer(open("output.csv", "w"))
    c = connection.cursor()

    rows = c.fetchall()

    for x in rows:
        csvWriter.writerows(x)

Answer 1

You should just do:

rows = c.fetchall()
csvWriter.writerows(rows)

If the reason you iterate through the rows is because you wan't to preprocess them before writing them to the file, then use the writerow method:

rows = c.fetchall()
for row in rows:
    # do your stuff
    csvWriter.writerow(row)

Answer 2

In order to put tittles in first row, dictionary approach is suggested for table inventory in cars.db

import sqlite3
import csv
import os.path
BASE_DIR = os.path.dirname(os.path.abspath(__file__))
db_path = os.path.join(BASE_DIR, "cars.db")
conn = sqlite3.connect(db_path)
c = conn.cursor()
c.execute("SELECT rowid, * FROM inventory")    
columns = [column[0] for column in c.description]
results = []
for row in c.fetchall():
    results.append(dict(zip(columns, row)))
with open("output.csv", "w", newline='') as new_file:
    fieldnames = columns
    writer = csv.DictWriter(new_file,fieldnames=fieldnames)
    writer.writeheader()
    for line in results:
        writer.writerow(line)
conn.close()

Answer 3

Using Pandas should be more performant and requires less code. You can save the data from a sqlite table to a Pandas DataFrame and then use Pandas to write the CSV file.

df = pd.read_sql('SELECT * from cars', conn)
df.to_csv('cars.csv')

Here's the full code that creates your sqlite table with fake data:

import pandas as pd
import sqlite3

# create Pandas DataFrame
data = [('Toyota', 'Hilux', 2),
    ('Ford', 'Tek', 6),
    ('Ford', 'Outlander', 9),
    ('Honda', 'Dualis', 3),
    ('Honday', 'Elantre', 4)]
df = pd.DataFrame.from_records(data, columns=['make', 'model', 'age'])

# establish sqlite connection
conn = sqlite3.connect('../tmp/cars.db')
c = conn.cursor()

# create sqlite table
c.execute('''CREATE TABLE cars (make text, model text, age int)''')

# add data to sqlite table
df.to_sql('cars', conn, if_exists='append', index = False)

# write sqlite table out as a CSV file
df = pd.read_sql('SELECT * from cars', conn)
df.to_csv('../tmp/cars.csv')

Here's code to write out all the tables in a sqlite database as CSV files with a single command:

for table in c.execute("SELECT name FROM sqlite_master WHERE type='table';").fetchall():
    t = table[0]
    df = pd.read_sql('SELECT * from ' + t, conn)
    df.to_parquet('../tmp/' + t + '.csv')

See here for more info.