I try to use parallelism with word2vec implemented in the gensim library. I notice that, more I increase of threads, more the training is slow and I don't know why. Are there any settings to make? I use : - debian - python 3.6.9 - cython how can i benefit parallelism ?
Thanks for advance
Gensim's default & original way of specifying a training corpus for Word2Vec (& related models) is via a single iterable object which can provide each text example in turn. Then, a single master thread reads from the iterable, parcelling out batches of texts to any number of worker threads (controlled by the workers parameter).
This still faces a few performance bottlenecks that prevent full utilization of large numbers of threads, especially as the number of threads grow.
First, if the iterable object is itself doing any time-consuming work to prepare each item – such as tokenization or preprocessing, or IO to a laggy/remote source – the single master thread may not send off texts as fast as the many workers can process them, becoming the limiting factor. (You can help this somewhat by ensuring your iteratable is doing the least amount of IO or regex/text-scanning/lookup possible - such as by using a corpus already tokenized in memory, or only reading an already tokenized/preprocessed corpus from disk requiring only simple item/token splitting on whitespace/linebreaks.)
Second, Python's 'Global Interpreter Lock' (GIL) means most pure-Python code can only be run by one thread at a time. Gensim uses Cython & library code to enable much of the workers' most-intense tasks to happen outside this GIL bottleneck, but some aspects of each thread's control-loops & inter-thread result handoffs still need the GIL. So As the number of worker threads grow, contention over the GIL becomes more of a limiting factor – and thus even with 16+ cores, training throughput often maxes somewhere around 5-12 threads. (Some parameters choices that intensify certain aspects of the training – like larger vectors sizes or more negative examples – can reduce the contention, but may not improve runtime, as those options just reclaim contended time for more calculation.)
Recent versions of gensim include an alternate method of supplying the corpus, if you can make your corpus available as a single file where each text is on its own line with all tokens separated by whitespace. In that case, every worker can open its own view onto a range of the file, allowing their training to proceed completely without the GIL/interthread handoffs.
To use this alternative, specify your corpus using the corpus_file parameter, as a path to the file.
This parameter is mentioned in the Word2Vec class docs and there's some more discussion of its usage in the release notes for gensim version 3.6.0.
With this option, training throughput should generally improve nearly linearly with each additional workers thread, up to the number of CPU cores available. (Note that the initial once-through vocabulary-building survey of the corpus is still single-threaded.)
