Spark SQL to explode array of structure

Question

I have the below JSON structure which I am trying to convert to a structure with each element as column as shown below using Spark SQL. Explode(control) is not working. Can someone please suggest a way to do this?

Input:

{
"control" : [
    {
      "controlIndex": 0,
      "customerValue": 100.0,
      "guid": "abcd",
      "defaultValue": 50.0,
      "type": "discrete"
    },
    {
      "controlIndex": 1,
      "customerValue": 50.0,
      "guid": "pqrs",
      "defaultValue": 50.0,
      "type": "discrete"
    }
  ]
}

Desired output:

controlIndex customerValue guid defaultValult  type

0            100.0         abcd   50.0         discrete

1            50.0          pqrs   50.0         discrete

I tried this - select explode(control) from myview It makes one column and puts each structure inside that — abiswas
– abiswas, Commented Oct 17, 2017 at 17:26

Hao Tan · Accepted Answer · 2018-10-15 02:43:15Z

3

Addition to Paul Leclercq's answer, here is what can work.

import org.apache.spark.sql.functions.explode   
df.select(explode($"control").as("control")).select("control.*")

answered Oct 15, 2018 at 2:43

Hao Tan

1,6363 gold badges14 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Paul Leclercq · Accepted Answer · 2017-10-17 20:31:34Z

1

Explode will create a new row for each element in the given array or map column

import org.apache.spark.sql.functions.explode   

df.select(
  explode($"control")
)

answered Oct 17, 2017 at 20:31

Paul Leclercq

1,0182 gold badges16 silver badges26 bronze badges

Comments

Kunal Jha · Accepted Answer · 2018-05-18 09:35:05Z

-1

Explode will not work here as its not a normal array column but an array of struct. You might want to try something like

df.select(col("control.controlIndex"), col("control.customerValue"), col ("control. guid"), col("control. defaultValue"), col(control. type))

answered May 18, 2018 at 9:35

Kunal Jha

74 bronze badges

Collectives™ on Stack Overflow

Spark SQL to explode array of structure

3 Answers 3

Comments

Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Linked

Related