Spark join 2 dataframe based on multiple columns

Question

I have 2 dataframes df1 and df2. I have join condition for these dataframes defined in a map. However the join column names are different.

I know that I can do like this, val df3 = df2.join(df1, df2("col1") <=> df1("col5") && df2("col2") <=> df1("col6") && df2("col3") <=> df1("col7") && df2("col4") <=> df1("col8"), "left" )

IS there a way to dynamically create the join condition using values in map.

Chema · Accepted Answer · 2020-06-18 22:04:42Z

2

You could use

val df3 = df2.join(df1, Seq("col5", "col6", "col7", "col8"), "left")

if you previously rename columns from df2 because columns must exist on both sides.

answered Jun 18, 2020 at 22:04

Chema

2,8382 gold badges17 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark join 2 dataframe based on multiple columns

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related