How to convert pyspark row to class?

Selva 0
Viewed 42

I have a dataframe like below,

df=spark.sql('select * from <table_name>')

++++++++++++++++++++++++++++++++++++++
|  Name    | Max   |  Name   | Avg   |
++++++++++++++++++++++++++++++++++++++
|  pouser1 | 1.0   |  2.0    | 3.0   |
|  pouser2 | 1.0   |  2.0    | 3.0   |
|  pouser3 | 1.0   |  2.0    | 3.0   |
+++++++++++++++++++++++++++++++++++++|

I have a class with below definition,

class pouser:

  @property
  def name():
     return self.__name

  @property
  def max():
     return self.__max

  @property
  def min():
     return self.__min

  @property
  def avg():
     return self.__avg

Is it possible to transform dataframe to list of pouser objects?

pyspark python
1 Answers

Yes it is possible (but is it really a use case for spark - maybe not).

Spark is a big data processing tool, where you can manipulate large data that cannot fit the computer memory. Your current example exactly does the opposite, trying to fit this data into computer memory in form of objects.

So if you still want to do that using spark you can collect the data into memory by using df.toPandas() and then create objects from there by iterating through the rows.

the_ordinary_guy0
Related