The goal is to perform with sampleBy or similar a sampling with weights greater than 1 (oversampling).
When using the following code I am getting the following error:
Fractions must be in [0, 1], but got Map(A -> 10.2, B -> 0.2, C -> 0.1, D -> 0.5)
Here is the example data:
import numpy as np
import pandas as pd
from pyspark.sql import functions as F
weights = {
"A":10.0,
"B":0.2,
"C":0.1,
"D":0.5
}
A_l = 90000
B_l = 4500466
C_l = 5243287
D_l = 144144
tot = A_l + B_l + C_l + D_l
df = pd.DataFrame({"hook":["A"]*A_l+["B"]*B_l+["C"]*C_l+["D"]*D_l,
"foo":["foo" for _ in range(tot)],
"bar":["bar" for _ in range(tot)],
"var":list(np.random.randint(2, size=tot)),
"var2":list(np.random.random(size=tot))
})
spark.conf.set("spark.sql.execution.arrow.enabled","true")
sparkdf=spark.createDataFrame(df)
Now I perform the sampleBy as I have a dict with the weights:
sparkdf.sampleBy(variable, fractions=weights, seed=0)