using map/reduce on lists of lists

user_4168794 0
Viewed 111

I have a very large list of lists, and I want to use map/reduce techniques (in Python/PySpark), in an efficient way, to calculate the PageRank of the network made of the elements in the list of lists that sharing a list means a link between them. I have no clue how to deal with the elements in the lists because considering all the possible pairs would be an unimaginably complex process.

Suppose this is the data (while it is a very huge list, sometimes with hundreds of elements in lists):

data = [[n1, n2], [n1, n3, n4, n5], [n2, n5, n7]]

For example, having something like the following would be much better than what I have:

n1 n2
n1 n3
n1 n4
n1 n5
n3 n4
n3 n5
n4 n5
n2 n5
n2 n7
n5 n7

Actually, I want to use the MapReduce technique to see how can I handle the situations like this in the future.

pyspark mapreduce python
2 Answers

First I thought of using map and then reduce for removing same pairs but below solution using itertools also seemed fine to me

import itertools
data = [["n1", "n2"], ["n1", "n3", "n4", "n5"], ["n2", "n5", "n7"]]
rd=sc.parallelize(data)
rd=rd.flatMap(lambda x:itertools.combinations(x,2))
rd.collect()

#output
Out[60]: [('n1', 'n2'),
 ('n1', 'n3'),
 ('n1', 'n4'),
 ('n1', 'n5'),
 ('n3', 'n4'),
 ('n3', 'n5'),
 ('n4', 'n5'),
 ('n2', 'n5'),
 ('n2', 'n7'),
 ('n5', 'n7')]
user_190757710

You can write a normal loop like this:

data = [[1, 2], [1, 3, 4, 5], [2, 5, 7]]

pairs = []
for di in data:
    for i, ai in enumerate(di[:-1]):
        for p in di[i+1:]:
            pairs.append([ai,p])

and print the pairs as np.array:

import numpy as np
print(np.array(pairs))
[[1 2]
 [1 3]
 [1 4]
 [1 5]
 [3 4]
 [3 5]
 [4 5]
 [2 5]
 [2 7]
 [5 7]]

If then you want to make it fast, you can use numba or JAX to jit-compile the function and achieve enormous speedups.

user_39431700
Related