Pass parameters to SQL in Databricks (Python)

Question

I have this field from a dataframe that I need to use as parameter for a SQl, so I'm using the code below and it throws empty, no error, just nothing.

d = df_max_dt.select('update_tstmp');
df_tmp = spark.sql("SELECT * FROM invoices WHERE last_upd_tstmp > from_utc_timestamp('{0}', 'MST') ".format(d);

However, if I manually replace the parameter like this below, it does work:

df_tmp = spark.sql("SELECT * FROM invoices WHERE last_upd_tstmp > from_utc_timestamp('{0}', 'MST') ".format('2020-02-25T18:54:48.435+0000'))

What am I missing here?

Check printing the "d" variable before assigning it to see if it has a value? — Armando
– Armando, Commented Mar 20, 2020 at 19:42

ywbaek · Accepted Answer · 2020-03-20 20:03:21Z

2

The issue is that d is a dataframe. Try this:

d = df_max_dt.select('update_tstmp').collect()[0].update_tstmp;
df_tmp = spark.sql("SELECT * FROM invoices WHERE last_upd_tstmp > from_utc_timestamp('{0}', 'MST') ".format(d);

answered Mar 20, 2020 at 20:03

ywbaek

3,0413 gold badges11 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pass parameters to SQL in Databricks (Python)

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related