How to convert a spark dataframe into a databrick koalas dataframe?

user_4564238 0
Viewed 8950

I know that you can convert a spark dataframe df into a pandas dataframe with

df.toPandas()

However, this is taking very long, so I found out about a koala package in databricks that could enable me to use the data as a pandas dataframe (for instance, being able to use scikit learn) without having a pandas dataframe. I already have the spark dataframe, but I cannot find a way to make it into a Koalas one.

databricks dataframe python-3.x
2 Answers

To go straight from a pyspark dataframe (I am assuming that is what you are working with) to a koalas dataframe you can use:

koalas_df = ks.DataFrame(your_pyspark_df)

Here I've imported koalas as ks.

user_74518110

Well. First of all, you have to understand the reason why toPandas() takes so long :

  • Spark dataframe are distributed in different nodes and when you run toPandas()

  • It will pull the distributed dataframe back to the driver node (that's the reason it takes long time)

  • you are then able to use pandas, or Scikit-learn in the single(Driver) node for faster analysis and modeling, because it's like your modeling on your own PC

  • Koalas is the pandas API in spark and when you convert it to koalas dataframe : It's still distributed, so it will not shuffle data between different nodes, so you can use pandas' similar syntax for distributed dataframe transformation
user_92267210
Related