In Pandas dataframe I can get the first 1000 rows with data.iloc[1:1000,:]. How I can do it in PySpark?
In Pandas dataframe I can get the first 1000 rows with data.iloc[1:1000,:]. How I can do it in PySpark?
You can use df.limit(1000) to get 1000 rows from your dataframe. Note that Spark does not have a concept of index, so it will just return 1000 random rows. If you need a particular ordering, you can assign a row number based on a certain column, and filter the row numbers. e.g.
import pyspark.sql.functions as F
df2 = df.withColumn('rn', F.row_number().over(Window.orderBy('col_to_order'))) \
.filter('rn <= 1000')