Add new column with its data to existing DataFrame using

Question

In scala I have a List[String] which I want to add as a new Column to an existing DataFrame.

Original DF:

Name  | Date
======|===========
Rohan | 2007-12-21
...   | ...
...   | ...

Suppose want to add a new Column of Department

Expected DF:

Name | Date       | Department
=====|============|============
Rohan| 2007-12-21 | Comp
...  | ...        | ...
...  | ...        | ...

How can I do this in Scala?

So you want to join them on? I wouldn't think you just want to add some random values. — Reactormonk
– Reactormonk, Commented Oct 3, 2016 at 10:16
@Reactormonk Just need to add a new data column not sure if I need to use joins — Osy
– Osy, Commented Oct 3, 2016 at 13:00

Sandeep Purohit · Accepted Answer · 2016-10-03 10:37:05Z

1

You can do it with one way like just create the dataframe of name and listvalues and join both the dataframe with name column

answered Oct 3, 2016 at 10:37

Sandeep Purohit

3,71222 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Osy · Accepted Answer · 2016-10-03 13:09:27Z

1

This solved my issue

val newrows = dataset.rdd.zipWithIndex.map(_.swap)
      .join(spark.sparkContext.parallelize(results).zipWithIndex.map(_.swap))
      .values
      .map { case (row: Row, x: String) => Row.fromSeq(row.toSeq :+ x) }

Still need some exact explanation of it.

answered Oct 3, 2016 at 13:09

Osy

1151 gold badge6 silver badges24 bronze badges

Collectives™ on Stack Overflow

Add new column with its data to existing DataFrame using

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related