In gensim Fasttext (or Word2vec), I would like to set a threshold for the vocabulary to the 100,000 most frequent words. How do I do this?

user_15739947 0
Viewed 186

I am using Fasttext (from Gensim). I have two issues I don't know how to solve:

  1. I would like to set a threshold for the vocabulary to the 100,000 most frequent words. 2. I would like to ensure that a list of words (from a text file) are part of the vocabulary as well. Say this list of words is in a text file called list.txt.

How would I do this?

Here is my model:

from gensim.models import FastText

class paragraph_generator(object):
    def __init__(self,test=True,itersize=10000,year=None,state=None):
        self.test=test
        self.itersize=itersize
        self.sql = f"""
        SELECT
            text_id,
            lccn_sn,
            date,
            ed,
            chroniclingamerica_meta.statefp,
            chroniclingamerica_meta.countyfp,
            text_ocr
        FROM
            chroniclingamerica natural join chroniclingamerica_meta
        WHERE date_part('year',date) BETWEEN 1870 AND 1920 
        AND  seq = 1 """
        if self.test:
            self.sql = self.sql+' limit 10000'   # limit 1000 means it only goes through 1000 lines of the database 
        else:
            pass
        print(self.sql)
    def __iter__(self):
        con, cur = database_connection.connect(cursor_type='server')
        cur.itersize = self.itersize
        cur.execute(self.sql)
        for p in cur.fetchall():
            tokens = p[-1].translate(str.maketrans('', '', punct)).replace('\n',' ').lower().split(' ')
            yield tokens
        con.close()


model = FastText(vector_size=256, window=8, min_count=10, epochs=1, workers=workers)
vocab = model.build_vocab(paragraph_generator(test=False, itersize=10000, year=None, state=None))
model.train(paragraph_generator(test=False, itersize=10000, year=None, state=None),
            total_examples=model.corpus_count, epochs=1)

I'm thinking of a mix between the parameters total_words and sorted_vocab, but I would not know how to do this.

Many thanks in advance for your answers!

word2vec fasttext gensim
2 Answers

Use the optional parameter max_final_vocab. From the Gensim FastText documentation:

max_final_vocab (int, optional) – Limits the vocab to a target vocab size by automatically selecting min_count. If the specified min_count is more than the automatically calculated min_count, the former will be used. Set to None if not required.

For example, change your line constructing the FastText instance to:

model = FastText(vector_size=256, window=8, 
                 max_final_vocab=100000, epochs=1, 
                 workers=workers)

A few other notes:

  • If this actually results in singleton words being taken, you still might want to set a min_count that's higher, and thus wind up with a smaller-than-100,000-word vocabulary. (Often these sorts of models do better if you're discarding the rarest words wiht one or only a handful of occurrences.)
  • Usually even if you have more CPU cores, you won't want workers to be more than some number in the 6-12 range. (If you need even more training speed, you could look into the alternate corpus_file option – you'd have to write your full corpus to a single large text file on disk, but then you could set workers up to the full number of cores – even 16, 32, or more – and continue to get high utilization.)

Finally, I haven't looked closely at your generator, but usually you'd want to make sure your corpus works as a Python sequence that is re-iterable. That is, downstream code and thus iterate over it multiple times. Then, you wouldn't be limited to epochs=1.

You could potentially check to make sure it is properly re-iterable with code like:

corpus = paragraph_generator(test=False, itersize=10000, year=None, state=None)
print(sum(1 for _ in corpus))
print(sum(1 for _ in corpus))

If that prints the same number twice – the actual expected number of texts in your corpus – all is well. If there's instead an error or 0, that corpus object is only capable of one iteration, and is then exhausted (non-restartable), so you'd want to tune your code further.

user_1302880

the easiest way to do this would be to sort your vocabulary by frequency, find the frequency of the 100,000th word, and make min_count be that frequency. You can do similar in preprocessing, and you can certainly be more efficient about it, but I think the most straightforward way is what I described above.

user_91554830
Related