I want to apply some custom partitioning when working with a given DataFrame. I found that the RDD groupBy provides me with the desired functionality. Now when I say
dataframe.rdd.groupBy(lambda row: row[1:3], numPartitions, partitioner)
I end up with a PythonRDD that has a tuple as a key and a ResultIterator as the value. What I want to do next is convert this back to a DataFrame since I want to use the apply function of GroupedData. I have attempted multiple things but have been unlucky so far.
Any help would be appreciated!