I am trying to convert a pyspark dataframe column from array of string to a string.
df:
text
[this, is, a, book, that, I, like]
I need:
text
"this, is, a, book, that, I, like"
Based on How to convert column of arrays of strings to strings?,
My py3 code:
import pyspark.sql.functions as F
t = df.withColumn('text', F.concat_ws(", ", df.text))
error:
ValueError: too many values to unpack (expected 2)
I missed something ?
thanks
You're intending to join an array of strings, not to concatenate multiple columns (which is what concat_ws expects).
The code you're looking for is F.array_join(df['text'], ', ')
df.withColumn('text2', f.array_join(df['text'], ', ')).show(truncate=False)
+----------------------------------+--------------------------------+
|text |text2 |
+----------------------------------+--------------------------------+
|[this, is, a, book, that, I, like]|this, is, a, book, that, I, like|
+----------------------------------+--------------------------------+
