Pyspark - How to get random values from a DataFrame column

user_7965204 0
Viewed 10954

I have one column in a DataFrame which I need to select 3 random values in Pyspark. Could anyone help-me, please?

+---+
| id|
+---+
|123| 
|245| 
| 12|
|234|
+---+

Desire:

Array with 3 random values get from that column: 

**output**: [123, 12, 234]
pyspark-sql spark-dataframe pyspark
2 Answers

You can order in random order using rand() function first:

 df.select('id').orderBy(rand()).limit(3).collect()

For more information on rand() function, check out pyspark.sql.functions.rand.

user_13824880

Here's another approach that's probably more performant.

You can fetch three random rows with this code:

df.rdd.takeSample(False, 3)

Here's how to create an array with three integers if you don't want an array of Row objects:

list(map(lambda row: row[0], df.rdd.takeSample(False, 3)))

df.select('id').orderBy(F.rand()).limit(3) will generate this this physical plan:

== Physical Plan ==
TakeOrderedAndProject(limit=3, orderBy=[_nondeterministic#38 ASC NULLS FIRST], output=[id#32L])
+- *(1) Project [id#32L, rand(-4436287143488772163) AS _nondeterministic#38]

This post discusses fetching random values from a DataFrame column in more detail.

user_11251590
Related