Reorder Key-Value Pairs using map() in Spark Scala

Question

What is the equivalent of the following pySpark code in Spark-Scala?

rddKeyTwoVal = sc.parallelize([("cat", (0,1)), ("spoon", (2,3))])
rddK2VReorder = rddKeyTwoVal.map(lambda (key, (val1, val2)) : ((key, val1) ,
val2))
rddK2VReorder.collect()
// [(('cat', 0), 1), (('spoon', 2), 3)] -- This is the output.

banjara · Accepted Answer · 2016-09-06 04:31:01Z

2

val rddKeyTwoVal = sc.parallelize(Seq(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map{case (key, (val1, val2)) => ((key, val1), val2)}
rddK2VReorder.collect

or

val rddKeyTwoVal = sc.parallelize(Seq(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map(r=> ((r._1, r._2._1),r._2._2))
rddK2VReorder.collect

output:

 Array(((cat,0),1), ((spoon,2),3))

Thanks @Alec for suggesting first approach

edited Sep 6, 2016 at 4:31

answered Sep 6, 2016 at 4:25

banjara

3,8983 gold badges42 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alec Over a year ago

Although rddKeyTwoVal.map { case (key, (val1, val2)) => ((key, val1), val2) } might be a cleaner translation of the lambda...

Shri Over a year ago

Thanks for the prompt input guys! Looks like we all found the same answer at the same time. :)

Shri · Accepted Answer · 2016-09-06 13:55:30Z

1

I found my own answer! Posting to help rest of the community. This is cleanest Scala version of the code I posted above. Produces the exact same output.

val rddKeyTwoVal = sc.parallelize(Array(("cat", (0,1)), ("spoon", (2,3))))
val rddK2VReorder = rddKeyTwoVal.map{case (key, (val1, val2)) => ((key, val1),val2)}

rddK2VReorder.collect()

//Use the following for a cleaner output. 
rddK2VReorder.collect().foreach(println)

Output:

// With collect() menthod.

Array[((String, Int), Int)] = Array(((cat,0),1), ((spoon,2),3))

// If you use the collect().foreach(println)
((cat,0),1)
((spoon,2),3)

edited Sep 6, 2016 at 13:55

answered Sep 6, 2016 at 4:35

Shri

595 bronze badges

2 Comments

Alec Over a year ago

This is exactly the same as @shekhar answer... In fact, slightly less correct, since technically you didn't print the output in your Python version, you just collected it.

Shri Over a year ago

Thanks. I have modified the answer.

Collectives™ on Stack Overflow

Reorder Key-Value Pairs using map() in Spark Scala

2 Answers 2

2 Comments

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Related