Fast search in a large text file

user_7428875 0
Viewed 840

I'm trying to write a search program in C# which will search for a string in a large text file(5GB). I've done a simple code shown below but the search results are time consuming and can take about 30 mins to complete. This is how my code looks like-

public List<string> Search(string searchKey)
{
    List<string> results = new List<string>();
    StreamReader fileReader = new StreamReader("D:\Logs.txt");
    while ((line = fileReader.ReadLine()) != null)
    {
        if (line.Contains(searchKey)
        {
            results.Add(line);
        }
    }
}

Although the code works, it runs very slowly and takes about 30 mins to complete. Can we do something to bring the search time under a minute?

full-text-search .net c#
2 Answers
user_128565490

File indexing is implemented in the library Bsa.Search.Core

You can implement your own version of reading from a file. FileByLinesRowReader - read file by line and add documents with externalId equal line number. FileDocumentIndex has been tested on wiki data json dictionary

  • .Net Core

  • .Net 472

         var selector = new IndexWordSelector();
     var morphology = new DefaultMorphology(new WordDictionary(), selector);
     var fileName = "D:\Logs.txt";

     // you can implement your own file reader, csv, json, or other
     var index = new FileDocumentIndex(fileName, new FileByLinesRowReader(null), morphology);

     // if index is already exist we skip file indexing
         if (!index.IsIndexed)
     index.Start();
     while (!index.IsReady)
     {
         Thread.Sleep(300);
     }

     var query = "("one" | two) ~50 ("error*")".Parse("*");
     var found = index.Search(new SearchQueryRequest()
     {
         Field = "*",
         Query = query,
         ShowHighlight = true,
     });
     // where ExternalId is line number from file  
     //found.ShardResult.First().FoundDocs.FirstOrDefault().Value.ExternalId
user_34867630
Related