Pyspark - error when save data into Hive table "unresolved operator 'InsertIntoTable HiveTableRelation'"

Question

I use the following:

• pyspark library , version 2.3.1
• python, version 2.7.1
• hadoop, version 2.7.3
• hive, version 1.2.1000.2.6.5.30-1
• spark version 2

My hive table looks following:

CREATE TABLE IF NOT EXISTS my_database.my_table
(
    division STRING COMMENT 'Sample column'
)

I want to save data into HIVE using pyspark. I use the following code:

spark_session = SparkSession.builder.getOrCreate()
hive_context = HiveContext(spark_session.sparkContext)
hive_table_schema = hive_context.table("my_database.my_table").schema
df_to_save = spark_session.createDataFrame([["a"],["b"],["c"]], schema=hive_table_schema)
df_to_save.write.mode("append").insertInto("my_database.my_table")

But the following error occur:

Traceback (most recent call last):
  File "/home/my_user/mantis service_quality_check__global/scripts/row_counts_preprocess.py", line 147, in <module> df_to_save.write.mode("append").insertInto(hive_table_row_counts_str)
  File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/readwriter.py", line 716, in insertInto
  File "/usr/hdp/current/spark2-client/python/lib/py4j-0.10.6-src.zip/py4j/java_gateway.py", line 1160, in __call__
  File "/usr/hdp/current/spark2-client/python/lib/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
  pyspark.sql.utils.AnalysisException: u"unresolved operator 'InsertIntoTable HiveTableRelation `my_database`.`my_table`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [division#14], false, false;;\n'InsertIntoTable HiveTableRelation `my_database`.`my_table`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [division#14], false, false\n+- LogicalRDD [division#2], false\n"

Please is there someone who cane help with this? I am stuck with this few days

Answer 1

I found the issue. SparkSession has to support hive. The method enableHiveSupport() has to be call when spark session is created.

Then creation of spark session will looks like following

spark_session = SparkSession.builder.enableHiveSupport().getOrCreate()