pyspark.sql.functions.arrays_overlap

pyspark.sql.functions.arrays_overlap(a1: ColumnOrName, a2: ColumnOrName) → pyspark.sql.column.Column

Collection function: returns true if the arrays contain any common non-null element; if not, returns null if both the arrays are non-empty and any of them contains a null element; returns false otherwise.

New in version 2.4.0.

Changed in version 3.4.0: Supports Spark Connect.

Returns

Column

a column of Boolean type.

Examples

>>> df = spark.createDataFrame([(["a", "b"], ["b", "c"]), (["a"], ["b", "c"])], ['x', 'y'])
>>> df.select(arrays_overlap(df.x, df.y).alias("overlap")).collect()
[Row(overlap=True), Row(overlap=False)]

pyspark.sql.functions.array_contains pyspark.sql.functions.array_join