pyspark.sql.functions.map_entries#

pyspark.sql.functions.map_entries(col)[source]#

Map function: Returns an unordered array of all entries in the given map.

New in version 3.0.0.

Changed in version 3.4.0: Spark Connect.

Parameters

colColumn or str: Name of column or expression

Returns

Column: An array of key value pairs as a struct type

Examples

Example 1: Extracting entries from a simple map

>>> from pyspark.sql import functions as sf
>>> df = spark.sql("SELECT map(1, 'a', 2, 'b') as data")
>>> df.select(sf.sort_array(sf.map_entries("data"))).show()
+-----------------------------------+
|sort_array(map_entries(data), true)|
+-----------------------------------+
|                   [{1, a}, {2, b}]|
+-----------------------------------+

Example 2: Extracting entries from a map with complex keys and values

>>> from pyspark.sql import functions as sf
>>> df = spark.sql("SELECT map(array(1, 2), array('a', 'b'), "
...   "array(3, 4), array('c', 'd')) as data")
>>> df.select(sf.sort_array(sf.map_entries("data"))).show(truncate=False)
+------------------------------------+
|sort_array(map_entries(data), true) |
+------------------------------------+
|[{[1, 2], [a, b]}, {[3, 4], [c, d]}]|
+------------------------------------+

Example 3: Extracting entries from a map with duplicate keys

>>> from pyspark.sql import functions as sf
>>> originalmapKeyDedupPolicy = spark.conf.get("spark.sql.mapKeyDedupPolicy")
>>> spark.conf.set("spark.sql.mapKeyDedupPolicy", "LAST_WIN")
>>> df = spark.sql("SELECT map(1, 'a', 1, 'b') as data")
>>> df.select(sf.map_entries("data")).show()
+-----------------+
|map_entries(data)|
+-----------------+
|         [{1, b}]|
+-----------------+
>>> spark.conf.set("spark.sql.mapKeyDedupPolicy", originalmapKeyDedupPolicy)

Example 4: Extracting entries from an empty map

>>> from pyspark.sql import functions as sf
>>> df = spark.sql("SELECT map() as data")
>>> df.select(sf.map_entries("data")).show()
+-----------------+
|map_entries(data)|
+-----------------+
|               []|
+-----------------+