concatenate all struct fields nested to array in spark

Question

My schema structure is following. I need to concatenate #VALUE,@DescriptionCode and @LanguageCode these are nested to an array.

root
 |-- partnumber: string (nullable = true)
 |-- brandlabel: string (nullable = true)
 |-- availabledate: string (nullable = true)
 |-- description: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- #VALUE: string (nullable = true)
 |    |    |-- @DescriptionCode: string (nullable = true)
 |    |    |-- @LanguageCode: string (nullable = true)

I have tried a lot but nothing work for me. I need following schema

root
 |-- partnumber: string (nullable = true)
 |-- brandlabel: string (nullable = true)
 |-- availabledate: string (nullable = true)
 |-- descriptions: array (nullable = true)
 |-- |--   element: string (containsNull = true)

Can you share what you have tried please ?

eliasah
– eliasah

2016-08-06 19:04:26 +00:00
Commented Aug 6, 2016 at 19:04 — eliasah
– eliasah, Commented Aug 6, 2016 at 19:04

Daniel de Paula · Accepted Answer · 2016-08-07 11:15:36Z

1

I believe you need to create an User Defined Function:

import org.apache.spark.sql.functions._

val func: (Seq[Row]) => Seq[String] = {
  _.map( 
    element =>
      element.getAs[String]("#VALUE") + 
      element.getAs[String]("@DescriptionCode") +
      element.getAs[String]("@LanguageCode")
  )
}

val myUDF = udf(func)

df.withColumn("descriptions", myUDF(col("description"))).drop(col("description"))

For more information about UDFs, you can read this article.

answered Aug 7, 2016 at 11:15

Daniel de Paula

18k9 gold badges75 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Kumar · Accepted Answer · 2018-04-06 05:26:22Z

    `root
     |-- partnumber: string (nullable = true)
     |-- brandlabel: string (nullable = true)
     |-- availabledate: string (nullable = true)
     |-- description: array (nullable = true)
     |    |-- element: struct (containsNull = true)
     |    |    |-- #VALUE: string (nullable = true)
     |    |    |-- @DescriptionCode: string (nullable = true)
     |    |    |-- @LanguageCode: string (nullable = true)
     |    |    |-- @Language: string (nullable = true)`

    suppose We want to concatenate 2 struct fields as one string separated by :,next 2 struct fields as another column.


root
 |-- partnumber: string (nullable = true)
 |-- brandlabel: string (nullable = true)
 |-- availabledate: string (nullable = true)
 |-- descriptions: array (nullable = true)
 |-- |--   element1: string (containsNull = true)
 |-- |--   element2: string (containsNull = true)

Collectives™ on Stack Overflow

concatenate all struct fields nested to array in spark

2 Answers 2

Comments

Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Related