oversampling pyspark using sampleBy() in Pyspark

Viewed 26

The goal is to perform with sampleBy or similar a sampling with weights greater than 1 (oversampling).

When using the following code I am getting the following error:

Fractions must be in [0, 1], but got Map(A -> 10.2, B -> 0.2, C -> 0.1, D -> 0.5)

Here is the example data:

import numpy as np
import pandas as pd
from pyspark.sql import functions as F

weights = {
      "A":10.0,
      "B":0.2,
      "C":0.1,
      "D":0.5
}

A_l = 90000
B_l = 4500466
C_l = 5243287
D_l = 144144

tot = A_l + B_l + C_l + D_l

df = pd.DataFrame({"hook":["A"]*A_l+["B"]*B_l+["C"]*C_l+["D"]*D_l,
                 "foo":["foo" for _ in range(tot)],
                 "bar":["bar" for _ in range(tot)],
                 "var":list(np.random.randint(2, size=tot)),
                 "var2":list(np.random.random(size=tot))
                  })

spark.conf.set("spark.sql.execution.arrow.enabled","true")
sparkdf=spark.createDataFrame(df) 

Now I perform the sampleBy as I have a dict with the weights:

sparkdf.sampleBy(variable, fractions=weights, seed=0)
0 Answers
Related