Fetch the line that has two specific keywords

user_4941764 0
Viewed 75

I have a list that contains pair of keywords ('k1', 'k2'). Here's a sample:

print (word_pairs)
--->[('salaire', 'dépense'), ('gratuité', 'argent'), ('causesmwedemwelamwemort', 'cadres'), ('caractèresmwedumwedispositif', 'historique'), ('psychomotricienmwediplôme', 'infirmier'), ('impôtmwesurmwelesmweréunionsmwesportives', 'compensation'), ('affichage', 'affichagemweopinion'), ('délaimweprorogation', 'défaillance'), ('créancemwenotion', 'généralités')]

I have a text file r_isa.txt (205MB) that contain words that share an "isa" relationship. Here's a sample, where \t represents a literal tab character:

égalité de Parseval\tformule_0.9333\tégalité_1.0
filiation illégitime\tfiliation_1.0
Loi reconnaissant l'égalité\tloi_1.0
égalité entre les sexes\tégalité_1.0
liberté égalité fraternité\tliberté_1.0

This basically means, "égalité de Parseval" isa "formule" with a score of 0.9333 and isa "égalité" with a score of 1. And so go on..

I want to know based on the r_isa file, if the keyword k1 isa k2, and if k2 is-a k1. On the output file, I want to save on each line the pair of words that do have the is-a relationship.

Here's what I did:

#Reading data as list
keywords = [line for line in open('version_final_PMI_espace.txt', encoding='utf8')]
keywords = ast.literal_eval(keywords[0])
word_pairs = []

for k,v in keywords.items():
    if v:
        word_pairs.append((k,v[0][0]))
len(list(set(word_pairs)))
 #####  


with open("r_isa.txt",encoding="utf-8") as readfile, open('Hyperonymy_file_pair.txt', 'w') as writefile:
    for line in readfile:
        firstfield = line.split('\t')[0].lower()
        for w in word_pairs:
            if w[0]==firstfield:
                if w[1] in line: 
                    writefile.write("".join(w[0]) + "\t"+"".join(w[1]) +"\n" )

This returns random pairs to me, for exemple:

salaire\targent
dépense\tcadres

unstead of ( in case of an existing isa relationship)

salaire\tdépense
causesmwedemwelamwemort\tcadres

Where did I go wrong ?

python-3.x list python
2 Answers

Updated Answer

The statement if w[1] in line: is highly suspect. See the following code for what I believe the logic should be. Since I don't have access to your files, I have turned readfile into a list of strings for testing purposes and instead of writing output to writefile, I am just printing some results. I have added some values to word_pairs and readfile so that I get some results. Also note that if you are converting the input file to lower case, then your word pairs must also be lower case.

This code checks if k1 isa k2 and if not, then checks if k2 isa k1.

word_pairs = [('égalité de parseval', 'égalité'), ('salaire', 'dépense'), ('gratuité', 'argent'), ('causesmwedemwelamwemort', 'cadres'), ('caractèresmwedumwedispositif', 'historique'), ('psychomotricienmwediplôme', 'infirmier'), ('impôtmwesurmwelesmweréunionsmwesportives', 'compensation'), ('affichage', 'affichagemweopinion'), ('délaimweprorogation', 'défaillance'), ('créancemwenotion', 'généralités')]
word_pairs2 = [(pair[1], pair[0]) for pair in word_pairs] # reverse the words
word_dict = dict(word_pairs) # create a dictionary for fast searching
word_dict2 = dict(word_pairs2)

readfile = [
    'égalité de Parseval\tformule_0.9333\tégalité_1.0',
    'filiation illégitime\tfiliation_1.0',
    'Loi reconnaissant l\'égalité\tloi_1.0',
    'égalité entre les sexes\tégalité_1.0',
    'liberté égalité fraternité\tliberté_1.0',
    'dépense\tsalaire_.9'
]

for line in readfile:
    fields = line.lower().split('\t')
    first_word = fields.pop(0)
    isa_word = word_dict.get(first_word, word_dict2.get(first_word)) # check k2 isa k1 if k1 isa k2 is false
    if isa_word is not None:
        for field in fields: # check each one
            fields2 = field.split('_')
            second_word, score = fields2
            if second_word == isa_word:
                print(first_word, second_word, score)

Prints:

égalité de parseval égalité 1.0
dépense salaire .9

If it is possible that k1 isa k2 and k2 isa k1, then you need the more general (but more complicated) code:

word_pairs = [('égalité de parseval', 'égalité'), ('salaire', 'dépense'), ('gratuité', 'argent'), ('causesmwedemwelamwemort', 'cadres'), ('caractèresmwedumwedispositif', 'historique'), ('psychomotricienmwediplôme', 'infirmier'), ('impôtmwesurmwelesmweréunionsmwesportives', 'compensation'), ('affichage', 'affichagemweopinion'), ('délaimweprorogation', 'défaillance'), ('créancemwenotion', 'généralités')]
word_pairs2 = [(pair[1], pair[0]) for pair in word_pairs] # reverse the words
word_dict = dict(word_pairs) # create a dictionary for fast searching
word_dict2 = dict(word_pairs2)

readfile = [
    'égalité de Parseval\tformule_0.9333\tégalité_1.0',
    'filiation illégitime\tfiliation_1.0',
    'Loi reconnaissant l\'égalité\tloi_1.0',
    'égalité entre les sexes\tégalité_1.0',
    'liberté égalité fraternité\tliberté_1.0',
    'salaire\tdépense_1.0',
    'dépense\tsalaire_.9'
]

for line in readfile:
    fields = line.lower().split('\t')
    first_word = fields.pop(0)
    # k1 isa k2?
    isa_word = word_dict.get(first_word)
    if isa_word is not None:
        for field in fields: # check each one
            fields2 = field.split('_')
            second_word, score = fields2
            if second_word == isa_word:
                print(first_word, second_word, score)
    # k2 isa k1?
    isa_word = word_dict2.get(first_word)
    if isa_word is not None:
        for field in fields: # check each one
            fields2 = field.split('_')
            second_word, score = fields2
            if second_word == isa_word:
                print(first_word, second_word, score)

Prints:

égalité de parseval égalité 1.0
salaire dépense 1.0
dépense salaire .9
user_28237190 
kw = [('salaire', 'dépense'),
      ('gratuité', 'argent'),
      ('causesmwedemwelamwemort', 'cadres'),
      ('caractèresmwedumwedispositif', 'historique'),
      ('psychomotricienmwediplôme', 'infirmier'),
      ('impôtmwesurmwelesmweréunionsmwesportives', 'compensation'),
      ('affichage', 'affichagemweopinion'),
      ('délaimweprorogation', 'défaillance'),
      ('créancemwenotion', 'généralités')]

lines_from_file = ['égalité de Parseval\tformule_0.9333\tégalité_1.0',
                   'filiation illégitime\tfiliation_1.0',
                   'Loi reconnaissant l\'égalité\tloi_1.0',
                   'égalité entre les sexes\tégalité_1.0',
                   'liberté égalité fraternité\tliberté_1.0',
                   'créancemwenotion\tgénéralités_1.0',
                   'généralités\tcréancemwenotion_1.0']

who_is_who_dict = {}
for line in lines_from_file:
    words = line.split('\t')
    key = words[0]
    other_words = [w.split('_')[0] for w in words[1:]]
    if key in who_is_who_dict:
        who_is_who_dict[key] = who_is_who_dict[key] + other_words
    else:
        who_is_who_dict[key] = other_words

pairs_to_write = []
for kw1, kw2 in kw:
    if (kw1 in who_is_who_dict and kw2 in who_is_who_dict[kw1]
            and kw2 in who_is_who_dict and kw1 in who_is_who_dict[kw2]):
        pairs_to_write.append((kw1, kw2))

print(pairs_to_write)

output :

[('créancemwenotion', 'généralités')]
user_140814740
Related