PySpark dataframe column to list

user_12519275 0
Viewed 20899

I am trying to extract the list of column values from a dataframe into a list

+------+----------+------------+
|sno_id|updt_dt   |process_flag|
+------+----------+------------+
| 123  |01-01-2020|     Y      |
+------+----------+------------+
| 234  |01-01-2020|     Y      |
+------+----------+------------+
| 512  |01-01-2020|     Y      |
+------+----------+------------+
| 111  |01-01-2020|     Y      |
+------+----------+------------+

Output should be the list of sno_id ['123','234','512','111'] Then I need to iterate the list to run some logic on each on the list values. I am currently using HiveWarehouseSession to fetch data from hive table into Dataframe by using hive.executeQuery(query)

pyspark
2 Answers

it is pretty easy as you can first collect the df with will return list of Row type then

row_list = df.select('sno_id').collect()

then you can iterate on row type to convert column into list 

sno_id_array = [ row.sno_id for row in row_list]

sno_id_array 
['123','234','512','111']

Using Flat map and more optimized solution

sno_id_array = df.select("sno_id ").rdd.flatMap(lambda x: x).collect()
user_121296980

You could use toLocalIterator() to create a generator over the column. Since you wanted to loop over the results afterwards, this may be more efficient in your case. Using a generator you don't create and store the list first, but when iterating over the columns you apply your logic immediately:

sno_ids = df.select('sno_id').toLocalIterator()
for row in sno_ids:
    sno_id = row.sno_id
    # continue with your logic
    ...

Alternative one-liner using a generator expression:

sno_ids = (row.sno_id for row in df.select('sno_id').toLocalIterator())
for sno_id in sno_ids:
    ...
user_43628780
Related