Pyspark Dataframe Ordering Issue

Question

I am having the data shown in the image:

I have been trying to partition and order this data frame in such a way so that we get the output as shown in the below image:

Have tried partitioning and sorting with different columns such as partitioning by id and date and ordering by id, date, and column3 then again tried with partitioning and ordering by id and date but in all the cases it shows the different output.

Can anyone help with this? I am struggling for the past week on this.

can you post expected output & post code what you have tried so far ? — s.polam
– s.polam, Commented May 25, 2021 at 6:46

dasilva555 · Accepted Answer · 2021-05-25 08:09:46Z

1

I am not sure if I understand correctly but that should work for your example:

df = df.orderBy(f.col('id').asc(),f.col('date').asc(),f.col('column3').desc())

answered May 25, 2021 at 8:09

dasilva555

1331 gold badge2 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark Dataframe Ordering Issue

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related