What is the most efficient way to randomly change values into null values in pyspark?

user_7298979 0
Viewed 582

Trying to figure out how to replace a specific column in Pyspark with null values randomly. So changing a dataframe such as this:

| A  | B  |
|----|----|
| 1  | 2  |
| 3  | 4  |
| 5  | 6  |
| 7  | 8  |
| 9  | 10 |
| 11 | 12 |

and randomly change 25% of the values in column 'B' to null values:

| A  | B    |
|----|------|
| 1  | 2    |
| 3  | NULL |
| 5  | 6    |
| 7  | NULL |
| 9  | NULL |
| 11 | 12   |
pyspark apache-spark python-3.x
1 Answers

thanks to @pault I was able to answer my own question using the question he posted that you can find here

Essentially I ran something like this:

import pyspark.sql.functions as f
df1 = df.withColumn('Val', f.when(f.rand() > 0.25, df1['Val']).otherwise(f.lit(None))

Which will randomly select values with the column 'Val' and make it into a None value

user_72989790
Related