How can I iterate through a column of a spark dataframe and access the values in it one by one?

user_9999035 0
Viewed 6273

I have spark dataframe Here it is

I would like to fetch the values of a column one by one and need to assign it to some variable?How can it be done in pyspark.Sorry I am a newbie to spark as well as stackoverflow.Please forgive the lack of clarity in question

pyspark apache-spark-sql
2 Answers
col1=df.select(df.column_of_df).collect()
list1=[str(i[0]) for i in col1]
#after this we can iterate through list (list1 in this case)
user_49104250

I don't understand exactly what you are asking, but if you want to store them in a variable outside of the dataframes that spark offers, the best option is to select the column you want and store it as a panda series (if they are not a lot, because your memory is limited).

from pyspark.sql import functions as F

var = df.select(F.col('column_you_want')).toPandas()

Then you can iterate on it like a normal pandas series.

user_89477390
Related