how to add a Incremental column ID for a table in spark SQL

Question

I'm working on a spark mllib algorithm. The dataset I have is in this form

Company":"XXXX","CurrentTitle":"XYZ","Edu_Title":"ABC","Exp_mnth":.(there are more values similar to these)

Im trying to raw code String values to Numeric values. So, I tried using zipwithuniqueID for unique value for each of the string values.For some reason I'm not able to save the modified dataset to the disk. Can I do this in any way using spark SQL? or what would be the better approach for this?

Sorry..I figured out with this thread stackoverflow.com/questions/33102727/… — KM-Yash
– KM-Yash, Commented Jul 14, 2016 at 19:11
Can you please delete your question (since it's a duplicate)? Thanks. — Jacek Laskowski
– Jacek Laskowski, Commented Jan 29, 2017 at 23:23

Community · Accepted Answer · 2022-09-16 23:02:46Z

19

Scala

import org.apache.spark.sql.functions.monotonically_increasing_id
val dataFrame1 = dataFrame0.withColumn("index",monotonically_increasing_id())

Java

 Import org.apache.spark.sql.functions;
Dataset<Row> dataFrame1 = dataFrame0.withColumn("index",functions.monotonically_increasing_id());

edited Sep 16, 2022 at 23:02

CommunityBot

11 silver badge

answered Aug 8, 2017 at 12:59

Yugerten

8881 gold badge13 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

how to add a Incremental column ID for a table in spark SQL

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related