How to Connect Teradata using Pyspark

Question

I am trying to connect teradata server through PySpark.

My CLI code is as below,

from pyspark.sql import SparkSession
spark=SparkSession.builder
                  .appName("Teradata connect")
                  .getOrCreate()
df = sqlContext.read
               .format("jdbc")
               .options(url="jdbc:teradata://xy/",
                        driver="com.teradata.jdbc.TeraDriver",
                        dbtable="dbname.tablename",
                        user="user1",password="***")
               .load()

Which is giving error,

py4j.protocol.Py4JJavaError: An error occurred while calling o159.load. : java.lang.ClassNotFoundException: com.teradata.jdbc.TeraDriver

To resolve this I think, I need to add jar terajdbc4.jar and `tdgssconfig.jar.

In Scala, to add jar we can use

    sc.addJar("<path>/jar-name.jar")

If I use the same for PySpark, I am having error,

AttributeError: 'SparkContext' object has no attribute 'addJar'.

or

AttributeError: 'SparkSession' object has no attribute 'addJar'

How can I add jar terajdbc4.jar and tdgssconfig.jar?

Answer 1

Try following this post which explains how to add jdbc drivers to pyspark.

How to add jdbc drivers to classpath when using PySpark?

The above example is for postgres and docker, but the answer should work for your scenario. Note, you are correct about the driver files. Most JDBC drivers are in a single file, but Teradata splits it out into two parts. I think one is the actual driver and the other (tdgss) has security stuff in it. Both files must be added to the classpath for it to work.

Alternatively, simply google "how to add jdbc drivers to pyspark".