Pyspark: How to filter 10000 random elements from spark dataframe

You can solve this using randomSplit. The result will be approximate though. You might not get exactly 10000 rows but it will be close enough

import pandas as pd
totalrows = 40000
df = pd.DataFrame([["name_"+str(i) for i in range(totalrows)], list(range(totalrows))]).transpose()
sdf = spark.createDataFrame(df)

def random_sampler(X, totalrows, nrows=100, seed = 42):
    split_ratios = [nrows/totalrows, 1-(nrows/totalrows)]
    random_sampled_data = X.randomSplit([split_ratio for split_ratio in split_ratios], seed=seed)
    return random_sampled_data[0]


random_sampler(sdf, totalrows, nrows=10000, seed = 42).count()
# 9952

you can achieve it by using randomSplit