TypeError: in method 'IndexIDMap_add_with_ids', argument 4 of type 'faiss::IndexIDMapTemplate< faiss::Index >::idx_t const *'

Viewed 1196

I'm trying to do semantic search with Pre trained bert models and transformers. I'm using Facebook AI library Faiss.

The code is :

encoded_data = model.encode(df.Plot.tolist())
encoded_data = np.asarray(encoded_data.astype('float32'))
index = faiss.IndexIDMap(faiss.IndexFlatIP(768))
index.add_with_ids(encoded_data, np.array(range(0, len(encoded_data))))
faiss.write_index(index, 'movie_plot.index')

The error it's returning is :

TypeError                                 Traceback (most recent call last)
<ipython-input-19-c09b9ccadf2a> in <module>
----> 1 index.add_with_ids(encoded_data, np.array(range(0, len(encoded_data))))
      2 faiss.write_index(index, 'movie_plot.index')

~\t5\lib\site-packages\faiss\__init__.py in replacement_add_with_ids(self, x, ids)
    233 
    234         assert ids.shape == (n, ), 'not same nb of vectors as ids'
--> 235         self.add_with_ids_c(n, swig_ptr(x), swig_ptr(ids))
    236 
    237     def replacement_assign(self, x, k, labels=None):

~\t5\lib\site-packages\faiss\swigfaiss.py in add_with_ids(self, n, x, xids)
   4950 
   4951     def add_with_ids(self, n, x, xids):
-> 4952         return _swigfaiss.IndexIDMap_add_with_ids(self, n, x, xids)
   4953 
   4954     def add(self, n, x):

TypeError: in method 'IndexIDMap_add_with_ids', argument 4 of type 'faiss::IndexIDMapTemplate< faiss::Index >::idx_t const *'

When i ran the same program in google colab, no error was returned. I'm running this program now in windows 10 local pc

I got the answer, we have to convert the np.array(range(0, len(encoded_data))) into int64

encoded_data = model.encode(df.Plot.tolist())
encoded_data = np.asarray(encoded_data.astype('float32'))
index = faiss.IndexIDMap(faiss.IndexFlatIP(768))
ids = np.array(range(0, len(df)))
ids = np.asarray(ids.astype('int64'))
index.add_with_ids(encoded_data, ids)
faiss.write_index(index, 'movie_plot.index')
2 Answers

you migth convert encoded_data.astype('float32') after having done np.asarray(encoded_data) such as:

np.asarray(encoded_data).astype('float32')
Related