Spark/Scala : Spark DataFrame column with Struct Type

Question

I have input like this .

Input:

|customerId|Header     |Line     |
|1001      |1001aa     |1001aa1  |
|1001      |1001aa     |1001aa2  |
|1001      |1001aa     |1001aa3  |
|1001      |1001aa     |1001aa4  |
|1002      |1002bb     |1002bb1  |
|1002      |1002bb     |1002bb2  |
|1002      |1002bb     |1002bb3  |
|1002      |1002bb     |1002bb4  |
|1003      |1003cc     |1003cc1  |
|1003      |1003cc     |1003cc2  |
|1003      |1003cc     |1003cc3  |
+----------+-----------+---------+

I want the output be of type

Using Dataframe and UDF I am able to do this But I would like to have those column names as well with struct Datatype. Any help is appreciated.

 val udfHeaderLineList1 = udf((header:String,line:Seq[String])=>{
  line.map(records=>List(header,records)).toList
})


val eventingDFtable = my_dataframe_data_Table.
  groupBy(col("customerId"), col("header")).
  agg(collect_list(col("Line")).alias("Line")).
 withColumn("TransHeaderStruct",udfHeaderLineList1(col("header"),col("Line"))).printSchema

Rohan Nayak · Accepted Answer · 2017-05-28 13:41:49Z

1

I got this solution by creating case class

case class simpleCaseClass (header:String,line:String)

val udfHeaderLineList3 = udf((header:String,line:Seq[String])=>{
      line.map(records=>List(header ,records)).map(value=>simpleCaseClass(value(0),value(1)))
    }

answered May 28, 2017 at 13:41

Rohan Nayak

2335 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Anahcolus Over a year ago

Can you check if your this solution matches with the desired schema in your question? I doubt

Rohan Nayak Over a year ago

Yes. It does matches. :)

Anahcolus Over a year ago

Yes indeed it does. ;) thanks . I guess you must have got your solution then. An upvote for your solution

Collectives™ on Stack Overflow

Spark/Scala : Spark DataFrame column with Struct Type

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related