String column doesn't exist in array column

Question

I would like to check if value from a string type column (not a string itself) doesn't exist in another array type column.

I tries something like this:

~col('string_column').isin(col('array_column'))

but since they are not the same type it fails

and also:

~col('array_column').contains(col('string_column'))

but this fails due to datatype mismatch.

How can I solve this?

mck · Accepted Answer · 2021-05-20 12:50:57Z

2

You can use array_contains with the columns:

import pyspark.sql.functions as F

df2 = df.withColumn('contain', ~F.expr('array_contains(array_column, string_column)'))

answered May 20, 2021 at 12:38

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

1 Answer 1