Array of String to Array of Struct in Scala + Spark

Question

I am currently using Spark and Scala 2.11.8

I have the following schema:

root
|-- partnumber: string (nullable = true)
|-- brandlabel: string (nullable = true)
|-- availabledate: string (nullable = true)
|-- descriptions: array (nullable = true)
|-- |--   element: string (containsNull = true)

I am trying to use UDF to convert it to the following:

root
|-- partnumber: string (nullable = true)
|-- brandlabel: string (nullable = true)
|-- availabledate: string (nullable = true)
|-- description: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- value: string (nullable = true)
|    |    |-- code: string (nullable = true)
|    |    |-- cost: int(nullable = true)

So source data looks like this:

[WrappedArray(a abc 100,b abc 300)]
[WrappedArray(c abc 400)]

I need to use " " (space) as a delimiter, but don't know how to do this in scala.

def convert(product: Seq[String]): Seq[Row] = {
    ??/
}

I am fairly new in Scala, so can someone guide me how to construct this type of function?

Thanks.

thopaw · Accepted Answer · 2017-02-18 07:46:51Z

2

I do not know if I understand your problem right, but map could be your friend.

case class Row(a: String, b: String, c: Int)
val value = List(List("a", "abc", 123), List("b", "bcd", 321))

value map {
    case List(a: String, b: String, c: Int) => Row(a,b,c);
}

if you have to parse it first:

val value2 = List("a b 123", "c d 345")
value2 map {
    case s => { 
        val split = s.toString.split(" ")
        Row(split(0), split(1), split(2).toInt)
    }
}

edited Feb 18, 2017 at 7:46

answered Feb 17, 2017 at 14:40

thopaw

4,0742 gold badges20 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

SuWon Over a year ago

Thank you for a quick reply. I try to implement your suggestion, but it is saying that "cannot resolve symbol split" Should I use different method?

thopaw Over a year ago

String.split is a method for strings. I dont know what type is in your wrapped arrays. Perhaps you have to call toString on each value to have a string. val split = s.toString.split(" ")

SuWon Over a year ago

Thanks. I was able to solve this problem with your help.

Collectives™ on Stack Overflow

Array of String to Array of Struct in Scala + Spark

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related