How I can get the same result using iloc in Pandas in PySpark?

Viewed 403

In Pandas dataframe I can get the first 1000 rows with data.iloc[1:1000,:]. How I can do it in PySpark?

1 Answers

You can use df.limit(1000) to get 1000 rows from your dataframe. Note that Spark does not have a concept of index, so it will just return 1000 random rows. If you need a particular ordering, you can assign a row number based on a certain column, and filter the row numbers. e.g.

import pyspark.sql.functions as F

df2 = df.withColumn('rn', F.row_number().over(Window.orderBy('col_to_order'))) \
        .filter('rn <= 1000')
Related