python regex unicode - extracting data from an utf-8 file

Viewed 830

I am facing difficulties for extracting data from an UTF-8 file that contains chinese characters.

The file is actually the CEDICT (chinese-english dictionary) and looks like this :

賓 宾 [bin1] /visitor/guest/object (in grammar)/
賓主 宾主 [bin1 zhu3] /host and guest/
賓利 宾利 [Bin1 li4] /Bentley/
賓士 宾士 [Bin1 shi4] /Taiwan equivalent of 奔馳|奔驰[Ben1 chi2]/
賓夕法尼亞 宾夕法尼亚 [Bin1 xi1 fa3 ni2 ya4] /Pennsylvania/
賓夕法尼亞大學 宾夕法尼亚大学 [Bin1 xi1 fa3 ni2 ya4 Da4 xue2] /University of Pennsylvania/
賓夕法尼亞州 宾夕法尼亚州 [Bin1 xi1 fa3 ni2 ya4 zhou1] /Pennsylvania/

Until now, I manage to get the first two fields using split() but I can't find out how I should proceed to extract the two other fields (let's say for the second line "bin1 zhu3" and "host and guest". I have been trying to use regex but it doesn't work for a reason I ignore.

#!/bin/python
#coding=utf-8
import re

class REMatcher(object):
    def __init__(self, matchstring):
        self.matchstring = matchstring

    def match(self,regexp):
        self.rematch = re.match(regexp, self.matchstring)
        return bool(self.rematch)

    def group(self,i):
        return self.rematch.group(i)

def look(character):
    myFile = open("/home/quentin/cedict_ts.u8","r")
    for line in myFile:
        line = line.rstrip()
        elements = line.split(" ")
        try:
            if line != "" and elements[1] == character:
                myFile.close()
                return line
        except:
            myFile.close()
            break
    myFile.close()
    return "Aucun résultat :("


translation = look("賓主") # translation contains one line of the file
elements = translation.split()
traditionnal = elements[0]
simplified = elements[1]
print "Traditionnal:" + traditionnal
print "Simplified:" + simplified

m = REMatcher(translation)
tr = ""
if m.match(r"\[(\w+)\]"):
    tr = m.group(1) 
print "Pronouciation:" + tr

Any help appreciated.

3 Answers
Related