What's the fastest way to copy files responsive to multiple search terms?

ghosj 0
Viewed 25

Presently working on an application that allows the user to input a list of names/search terms and an folder path. The application then searches for each phrase and copies any responsive documents to an output path. Important to note that the use is often with directories containing from 100GB up to a few TB and can sometimes be required to run thousands of search terms.

Initially I simply used the System.IO.GetFiles() function for this, but I've found I have better results creating a data table of all documents in the input path and running my searches over that data table (see below).

//Constructing a data table of all files in the input path
foreach (var file in fileArray)
    {
        System.Data.DataRow row = searchTable.NewRow();
        row[1] = file;
        row[0] = System.IO.Path.GetFileName(file);
        searchTable.Rows.Add(row);
    }
 
//For each line inputted by the user, search the data table to find any responsive file names
foreach (var line in searchArray)
{
    
    for (int i = 0; i < searchTable.Rows.Count; i++)
    {
        if (searchTable.Rows[i][0].ToString().Contains(line))
        {
            string file = searchTable.Rows[i][1].ToString();
            string output = SwiftBank.CalculateOutputFilePath(outputPath,inputPath,file);
            System.IO.File.Copy(file, output);
        }
    }
}

I've found that while this works, it isn't optimised and functions very slowly for large data sets. Obviously doing a lot of repeat work, searching the data table in full every search term. Wondering if someone on here might have a better idea?

datatable file io c# copy
1 Answers

In my experience, doing a handful of contains queries for a few thousands fairly short strings should take less than a second. If you are searching inside much larger data sets, like searching thru 100Gb of content, you should look at some more advanced library, like lucene.

There are a few things I would suggest changing

  • Use a regular list instead of a datatable. Something like List<(string filePath, string fileName)> would be much simpler, and contain the same information
  • Perform all checks for a specific file at once, i.e. reorder your loops the the file loop is the outer one. This should help cache usage a little bit.

However, the vast majority of the time will likely be spent on copying files. This is many orders of magnitude slower than doing some simple searching in a few kilobytes of memory. You might gain a little bit by doing more than one copy in parallel, since SSDs may be able to improve throughput at higher loads, but that is likely only true if the files are small. You might consider alternatives to the copying, like adding shortcuts, instead.

JonasH0
Related