How to convert single row of a dataframe into a String in Scala?

Question

My dataframe which was created by reading a RDBMS table, has one column and only one value in it:

val sourceCols = spark.read.format("jdbc").option("url", hiveMetaConURL)
                                               .option("dbtable", "(select source_columns from base.spe_tab where tablename='base.forecast') as sCols")
                                               .option("user", metaUserName)
                                               .option("password", metaPassword)
                                               .load()

I tried it convert it to a String in the below way:

val sourceColDataTypes = sourceCols.rdd.map(_.mkString(",")).collect.foreach(println)

When I try to print it as:

sourceColDataTypes.foreach(println)

I don't see the content, instead I see:

[Ljava.lang.String;@1e489957

Is there a way I can use yield of Scala to get the value. Could anyone let me know how can I convert a row in a DataFrame to a String ?

you should be doing sourceCols.rdd.map(_.toSeq.mkString(",")).collect.foreach(println) — Anahcolus
– Anahcolus, Commented Jul 28, 2018 at 9:39
The foreach(println) is printing the value. But if I assign it to val data:String = sourceCols.rdd.map(_.toSeq.mkString(",")).collect.toString() and print "data", it still prints java.lang.String@30a6984c — Metadata
– Metadata, Commented Jul 28, 2018 at 9:52

koiralo · Accepted Answer · 2018-07-28 09:43:06Z

1

To get the value you can use one of the following

sourceCols.map(_.getString(0)).collect.foreach(println)
sourceCols.map(_.toSeq.mkString(",")).collect.foreach(println)

If you just want to see the data you can use

sourceCols.show(false)

If you want a single string then you can get it as

println(sourceCols.map(_.getString(0)).collect.mkString(","))

Hope this helps!

answered Jul 28, 2018 at 9:43

koiralo

23.2k6 gold badges57 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Manoj Kumar Dhakad · Accepted Answer · 2018-07-28 18:31:20Z

0

You can try this directly on dataframe, There is no need to covert to rdd at all like this

df.select("value").collect.mkString(",").replaceAll("[\\[\\]]","")

Or

df.map(row=>row.getAs("value").toString).collect.mkString(",")

answered Jul 28, 2018 at 18:31

Manoj Kumar Dhakad

1,9021 gold badge14 silver badges27 bronze badges

Collectives™ on Stack Overflow

How to convert single row of a dataframe into a String in Scala?

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related