Pyspark crossJoin with specific condition

Viewed 1545

The crossJoin of two dataframes of 5 rows for each one gives a dataframe of 25 rows (5*5). What I want is to do a crossJoin but which is "not full". For example:

df1:                       df2:
+-----+                    +-----+
|index|                    |value|
+-----+                    +-----+
|    0|                    |    A|
|    1|                    |    B|
|    2|                    |    C|
|    3|                    |    D|
|    4|                    |    E|
+-----+                    +-----+

The result must be a dataframe of number of rows < 25, while for each row in index choosing randomly the number of rows in value with which the crossJoin is done. It will be something like that:

+-----+-----+
|index|value|
+-----+-----+
|    0|    D|
|    0|    A|
|    1|    A|
|    1|    D|
|    1|    B|
|    1|    C|
|    2|    A|
|    2|    E|
|    3|    D|
|    4|    A|
|    4|    B|
|    4|    E|
+-----+-----+

Thank you

1 Answers
Related