I am trying to count the frequent itemset from a database. The database has 31101 baskets and 12592 items. I am able to get frequent items that occur more than some threshold. The element has a string data type (eg. 'FRO11987')
Now I am trying to form the pair from these frequent items using the cartesian operation. I get 735 frequent items and when I use the cartesian operation, I expected it to be a fast operation.
However, it is taking too long (more than 20 mins) to form the pair. I must be doing something wrong in writing the pyspark code. Any help on this appreciated. Thank you.
def get_sup(x):
x_sup = 0
for t in shared_itemset.value:
if x.issubset(t):
#if set({x}).issubset(set(t)):
x_sup +=1
if x_sup >= sup:
return x, x_sup
else:
return ()
data = sc.textFile('browsing.txt')
itemset = data.map(lambda line: ([item for item in line.strip().split(' ')]))
shared_itemset = sc.broadcast(itemset.map(lambda x: frozenset(x)).collect())
c1 = data.flatMap(lambda line: line.strip().split(' ')).distinct()
c1 = c1.map(lambda x : frozenset({x}))
print(c1.count())----------------------------------------------- 12592
f1 = c1.map(get_sup).filter(lambda x: x)
print(f1.count())----------------------------------------------- 735
f1m = f1.map(lambda x: tuple(x[0]))
f1c = f1m.cartesian(f1m).map(lambda x: frozenset(x[0]+x[1])).filter(lambda x: len(x)>1).distinct()
print(f1c.count())
The sample data is given below:
FRO11987 ELE17451 ELE89019 SNA90258 GRO99222
GRO99222 GRO12298 FRO12685 ELE91550 SNA11465 ELE26917 ELE52966 FRO90334 SNA30755 ELE17451 FRO84225 SNA80192
ELE17451 GRO73461 DAI22896 SNA99873 FRO86643
ELE17451 ELE37798 FRO86643 GRO56989 ELE23393 SNA11465
ELE17451 SNA69641 FRO86643 FRO78087 SNA11465 GRO39357 ELE28573 ELE11375 DAI54444
I am able to form the frequent items and pair from the above data smoothly. The problem starts happening once the data is large (with 31101 baskets). For the whole dataset use this file