I have code like this:
def f(items):
results = []
for item in items:
results.append(process(item))
return results
rdd.mapPartitions(f)
But this requires all results to be stored in memory, instead of processed one row at time.
In regular spark I know you're supposed to do an "iterator-to-iterator" transform here (according to the "High Performance Spark" book). Can I achieve the same result in pyspark with a generator like this?
def f(items):
results = []
for item in items:
yield(process(item))
rdd.mapPartitions(f)