How to create a PySpark DataFrame from 2 columns of 2 DataFrames?

Question

I have the following 2 PySpark DataFrames, both with the same number of rows (say 100 rows):

df1:
 |_ Column_a
 |_ Column_b

df2:
 |_ Column_c
 |_ Column_d

How do I create df_final which has 100 rows and the following columns?:

df_final:
 |_ Column_a
 |_ Column_b
 |_ Column_c
 |_ Column_d

I looked at concat(), join(), union() but I don't think that's right.

Answer 1

Try zip

>>> df1.show()
+---+---+
|  a|  b|
+---+---+
|  2|  3|
|  4|  5|
+---+---+

>>> df2.show()
+---+---+
|  c|  d|
+---+---+
| 20| 30|
| 40| 50|
+---+---+

>>> df1.rdd.zip(df2.rdd).map(lambda x: (x[0][0],x[0][1],x[1][0],x[1][1])).toDF(['a','b','c','d']).show()
+---+---+---+---+
|  a|  b|  c|  d|
+---+---+---+---+
|  2|  3| 20| 30|
|  4|  5| 40| 50|
+---+---+---+---+