Working with very large Dictionary in C#

Question

I am implementing a type of search (TF-IDF) in which every word has a score calculated that is proportional to all the documents being searched. I have 100GB of documents to search.

If I was working with 1GB of documents, I would use:

Dictionary>

..where string is the word and List is all documents, ranked in order, containing that word. This does not scale up. I am using a Dictionary<> because lookup time is O(1) (in theory).

My intended solution is a SQLServer database in which words are listed in a table, with the relevant List object stored serialized. My concern is that reading the DB and rebuilding to List<> each time will be very inefficient.

Am I going in the wrong direction here? What is a normal solution to working with huge dictionaries?

user_12000637 · Answer

I am only touching on how you should store/retrieve the data. I would try something like Dictionary where List holds the ids of the documents you are referring to. Your database would have a Word table a WordToDocument table and a Document table.

If you are building a website there is no reason to return the entire contents of every document. You would only need to return the name of the documents as a user could them select the document they would like to read.

Working with very large Dictionary<> in C#

4 Answers