i am trying to do a recursive function that allows me to do the following algorithm.
I have a dataframe like this:
+----+---+
|Prod| No|
+----+---+
| A| 1|
| A| 2|
| B| 1|
| B| 5|
| 1| x|
| B| 2|
| 2| z|
| x| w|
+----+---+
And the output I need is:
+----+---+
|Prod| No|
+----+---+
| A| A|
| A| 1|
| A| 2|
| A| x|
| A| w|
| B| B|
| B| 1|
| B| 5|
| B| x|
| B| w|
| B| 2|
| B| z|
| 1| 1|
| 1| x|
| 1| w|
| 2| 2|
| 2| z|
| x| x|
| x| w|
+----+---+
That is, for each element on Prod column, i need all dependencies it has, directly or indirectly. Each element also has a dependence on each own.
I am trying a recursive function, but i'm loosing some elements. I have converted dataframes to list for easy operation. Can anyone help? Thank you.
The code i have is like this:
def check_dependencies(value):
next_list = df.where(df.Prod == value).select('No').collect()
return next_list
def recursive (dependencia):
global new_iter
dep_dinam = dependencia
if(new_iter):
k=i
else:
k=0
newiter= False
j=i
comprimento = len(dep_dinam)
#for k in range(len(dependencia)):
while k < comprimento:
sub = check_dependencies(dep_dinam[k][0])
list.extend(sub)
if sub == []:
lista_final.extend(list)
return
else:
dep_dinam = sub
comprimento = len(sub)
k = k+1
recursive(dep_dinam)
for i in range(len_unicos):
print(lista_unicos[i][0])
new_iter=True
recursive(lista_unicos)
Before, i have defined
data = [('A','1'),('A','2'),('A','3'),('B','1'),('B','5'),('1','x'),('1','y'),('B','2'),('2','z'),('x','w')]
columns = ["Prod","No"]
df = spark.createDataFrame(data=data, schema = columns)
unicos = df.select(df.Prod).distinct()
lista_unicos = unicos.collect()
In the example i have a smaller dataframe just for presentation purpose.