I'm developing an NLP search engine based on vectors and the MUSE (Multilingual Universal Sentence Encoder) model from Google.
The process is very simple:
1 - I have a folder with 34 pdf documents (190 MB)
2 - I convert this documents to plain text
3 - I tokenize the document´s plain text to sentences
4 - I use MUSE to encode every sentence in a 512 dimension vector
5 - I store the sentences + the vectors in elasticsearch, the vectors are stored in the "dense_vector" field
6 - When I do the search, I vectorize the search input (with MUSE, obviously) and I use the cosineSimilarity function from elastic to get the most relevant sentences.
So far so good, the search engine works with very decent results.
The problem here is the size of the index in elasticsearch, to store the 34 documents (190MB) took 560MB!, the index with the sentences + vectors is almost 3 times the size of the original documents in pdf.
I have read a lot of publications about the state of the art in NLP search engines and all of them point to sentence encoding (embeddings) and calculate the cosin to get the most relevant results, but any of them talks about a real world implementation.
Any of you have experience with something similar?, how do you deal with vectors storage?
Thanks!