Efficiently printing the intersection and union of huge different lists of strings

user_7294253 0
Viewed 213

I have three lists of differents sentences :

the first has (6228 elements), the second (30177) elements and the last (1059).

The string in each list correspond to sentences betwen 3 to more than 150 characters. Those sentences were extracted and seperated in different list in order to write them in different .tsv files. So Now I want to search in them to find the common sentences to the three lists and also the intersection between them.

I would like to print the intersection of the three lists with and without doublon (because somme elt in list_1 can be present in list_2 or list_3. I want also to print the intersection between each list with the other one :

for example intersection of element in list 1 and 2 but not in list 3 ; element in list 2 & 3 but not in list 1; and element in list 3 & 1 but not in list 2 ;

I come up with this algorythm but is there any better way to get what I want ? I ask because soince I amdealing with long string(sentences) I am afraid sometimes the looping in the list is not really efficient and do not brought me accurate result. Since my files are huges. I give you this small sample

    l_1 = ['je mange du pain', 'tu es laid', 'je suis beau', 'vive la vie', 'vive l\'horizon']
    l_2 =  ['je mange du pain', 'vive l\'horizon', 'L\'esprit des eaux est vaincu', 'Satan tu es vaincu', 'Jésus est puissant et t\'a vaincu', 'Satan tu n\'es rien' , 'tu ne peux pas m\'intimider', 'Je suis couverte par le Sang de Jésus']
    l_3 = ['je mange du pain', 'vive l\'horizon', 'L\'esprit des eaux est vaincu', 'Je suis couverte par le Sang de Jésus', 'vive la vie']

    sentences_1 = []    # intersection
    sentences_2_1 = []   # intersection between two list without checking if the elt is present the last list
    sentences_2_2 = []
    sentences_2_3 = []

    sentences_2_1_0 = []  intersection between two list with checking if the elt is present the last list
    sentences_2_2_0 = []
    sentences_2_3_0 = []
    sentences_3 = []      # Union of the three list (can contains sentences which are present more than once)

    #  One case
    for elt in l_2:
      if elt in l_1 and elt in l_3 :
        if elt not in sentences_1:
          sentences_1.append(elt)

    print('Phrases communes aux trois conditions :', len(sentences_1))


    # Second case 
    for elt in l_1:     
      if elt in l_2:
        if elt not in sentences_2_1_0:
          sentences_2_1_0.append(elt)

    print('Phrases communes aux sets 1 & 2 :', len(sentences_2_1_0))

    for elt in l_1:     
      if elt in l_2 and elt not in l_3:
        if elt not in sentences_2_1:
          sentences_2_1.append(elt)

    print('Phrases communes aux sets 1 & 2 mais pas 3:', len(sentences_2_1))


    # Second case 
    for elt in l_2:     
      if elt in l_3:
        if elt not in sentences_2_2_0:
          sentences_2_2_0.append(elt)

    print('Phrases communes aux sets 2 & 3 :', len(sentences_2_2_0))

    for elt in l_2:      
      if elt in l_3 and elt not in l_1 :
        if elt not in sentences_2_2:
            sentences_2_2.append(elt)

    print('Phrases communes aux sets 2 & 3 mais pas 1:', len(sentences_2_2))


    # Second case 
    for elt in l_3:     
      if elt in l_1 :
        if elt not in sentences_2_3_0:
          sentences_2_3_0.append(elt)

    print('Phrases communes aux sets 3 & 1 :', len(sentences_2_3_0))  

    for elt in l_1:
      if elt in l_3 and elt not in l_2 :
          if elt not in sentences_2_3:
            sentences_2_3.append(elt)

    print('Phrases communes aux sets 3 & 1 mais pas 2:', len(sentences_2_3))

    # third case 
    for elt in l_1:
      sentences_3.append(elt)

    for elt in l_2:
      sentences_3.append(elt)

    for elt in l_3: 
      sentences_3.append(elt)  
      #any(x, y, z) not in sentences_3:



    print('Union des trois sets:', len(sentences_3))

    sentences_3_filtered = sorted(list(set(sentences_3)))  # Print the union of the three list without doublon, we keep each sentences just once deleting doublons or triple

    print( "sans doublon :", len(sentences_3_filtered))
list-comprehension python-3.x python
2 Answers

Even a list of 30177 items (each a 150 character string) isn't very big.

You can just use sets directly, and do whatever intersection logic you'd like:

a = set([<items>...])
b = set([<items>...])
c = set([<items>...])

and then, e.g.:

a_and_b = a & b
a_and_c_but_not_b = a & c - b  # order of operations wouldn't matter anyway

should all be very fast.

If you have a truly large number of truly very large lists, sorting them first externally and then doing merge operations can provide some benefits.

user_22844900

There are 2 ways to do this: First either use a set or map. set would be wiser choice as it will save you space. Secondly if data you are processing is too big than you might want to consider the parallel procession by dividing it int equal parts.

user_154465570
Related