How to add multiple columns in Apache Spark

Question

Here is my input data with four columns with space as the delimiter. I want to add the second and third column and print the result

sachin 200 10 2
sachin 900 20 2
sachin 500 30 3
Raju 400 40 4
Mike 100 50 5
Raju 50 60 6

My code is in the mid way

from pyspark import SparkContext
sc = SparkContext()
def getLineInfo(lines):
    spLine = lines.split(' ')
    name = str(spLine[0])
    cash = int(spLine[1])
    cash2 = int(spLine[2])
    cash3 = int(spLine[3])
    return (name,cash,cash2)
myFile = sc.textFile("D:\PYSK\cash.txt")
rdd = myFile.map(getLineInfo)
print rdd.collect()

From here I got the result as

[('sachin', 200, 10), ('sachin', 900, 20), ('sachin', 500, 30), ('Raju', 400, 40
), ('Mike', 100, 50), ('Raju', 50, 60)]

Now the final result I need is as below, adding the 2nd and 3rd column and display the remaining fields

sachin 210 2
sachin 920 2
sachin 530 3
Raju 440 4
Mike 150 5
Raju 110 6

Thiago Baldim · Accepted Answer · 2016-09-08 13:32:57Z

4

Use this:

def getLineInfo(lines):
    spLine = lines.split(' ')
    name = str(spLine[0])
    cash = int(spLine[1])
    cash2 = int(spLine[2])
    cash3 = int(spLine[3])
    return (name, cash + cash2, cash3)

answered Sep 8, 2016 at 13:32

Thiago Baldim

7,7623 gold badges34 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Sachin Sukumaran Over a year ago

It Worked . Thanks . One more question - Once we have data in RDD , can we do some manipulation on specific data Example: I have in RDD [('sachin', 200, 10), ('Raju', 900, 20)] . How to Fetch 3rd column only , I mean 10 and 20

Rakesh Rakshit Over a year ago

@SachinSukumaran You should accept the answer if it worked.

Thiago Baldim Over a year ago

To fetch columns in different rows. I recommend you to read about: reduceByKey() method.

Collectives™ on Stack Overflow

How to add multiple columns in Apache Spark

1 Answer 1

3 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Linked

Related