I am using BERTopic to perform the topic modelling on 174,827 rows by using the following commands:
from bertopic import BERTopic
topic_model = BERTopic(language="english", calculate_probabilities=False, verbose=True)
topics, probs = topic_model.fit_transform(docs)
And it is giving me this following error:
Batches: 100%
5464/5464 [02:11<00:00, 90.71it/s]
2021-11-22 09:36:23,059 - BERTopic - Transformed documents to Embeddings
2021-11-22 09:43:58,215 - BERTopic - Reduced dimensionality with UMAP
---------------------------------------------------------------------------
_RemoteTraceback Traceback (most recent call last)
_RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/joblib/externals/loky/process_executor.py", line 407, in _process_worker
call_item = call_queue.get(block=True, timeout=timeout)
File "/usr/lib/python3.7/multiprocessing/queues.py", line 113, in get
return _ForkingPickler.loads(res)
File "sklearn/neighbors/_binary_tree.pxi", line 1057, in sklearn.neighbors._kd_tree.BinaryTree.__setstate__
File "sklearn/neighbors/_binary_tree.pxi", line 999, in sklearn.neighbors._kd_tree.BinaryTree._update_memviews
File "stringsource", line 658, in View.MemoryView.memoryview_cwrapper
File "stringsource", line 349, in View.MemoryView.memoryview.__cinit__
ValueError: buffer source array is read-only
"""
The above exception was the direct cause of the following exception:
BrokenProcessPool Traceback (most recent call last)
<ipython-input-9-ab3893bf488b> in <module>()
2
3 topic_model = BERTopic(language="english", calculate_probabilities=False, verbose=True)
----> 4 topics, probs = topic_model.fit_transform(docs)
10 frames
hdbscan/_hdbscan_boruvka.pyx in hdbscan._hdbscan_boruvka.KDTreeBoruvkaAlgorithm.__init__()
hdbscan/_hdbscan_boruvka.pyx in hdbscan._hdbscan_boruvka.KDTreeBoruvkaAlgorithm._compute_bounds()
/usr/lib/python3.7/concurrent/futures/_base.py in __get_result(self)
382 def __get_result(self):
383 if self._exception:
--> 384 raise self._exception
385 else:
386 return self._result
BrokenProcessPool: A task has failed to un-serialize. Please ensure that the arguments of the function are all picklable.
Meanwhile, if I do the same with approximately 50,000, it works completely fine. Can please someone help me understand how to get rid of this problem and make it work? I am using google colab with GPU.