Large dataframe generation in pyspark

user_14413902 0
Viewed 517

Cannot find solution in internet.

Try to create dataframe with 10 int columns and n rows of random values. Need to generate big amount of rows (n = 10 millions for example). If I have 10 rows of generated data (pyspark dataframe) by myself how can I populate them till n rows passed.
Generated dataframe should be written in parquet by pyspark.

What is the best solution for this?

pyspark dataframe python
1 Answers

You can use flatMap in a loop to create an exponential growing amount of rows:

rdd = spark.sparkContext.parallelize([(1,2,3,4,5,6,7,8,9,10)])

def f(t):    
    for c in range(0,10):        
        yield tuple((i+c) * 1664525 for i in t)

#Increase the size of this loop to create more data.
#The number of rows will be 10 ^ n
for _ in range(0, 2):
    rdd = rdd.flatMap(f)
    rdd = rdd.repartition(int(spark.conf.get('spark.sql.shuffle.partitions')))
    print(rdd.count())

#write result to parquet file
df = spark.createDataFrame(rdd)
df.write.parquet("mytestdata")
user_21298010
Related