Monitor folder and copy new files once their creation is complete

Question

I built a console app that monitors a set of folders on a Windows 2019 Server and copies any newly-created .txt files to another folder, using the same file name. So far it's working for the basic functionality. Now I have to handle the fact that most of the time, these files are large and take several minutes to complete creation. I have gone through several SO posts and pieced together the following code trying to accomplish this:

using System;
using System.IO;

namespace Folderwatch
{
    class Program
    {
        static void Main(string[] args)
        {
            string sourcePath = @"C:\Users\me\Documents\SomeFolder";

            FileSystemWatcher watcher = new FileSystemWatcher(sourcePath);

            watcher.EnableRaisingEvents = true;
            watcher.IncludeSubdirectories = true;
            watcher.Filter = "*.txt";

            // Add event handlers.
            watcher.Created += new FileSystemEventHandler(OnCreated);
        }

        // Define the event handlers. 

        private static void OnCreated(object source, FileSystemEventArgs e)
        {
            // Specify what is done when a file is created.
            FileInfo file = new FileInfo(e.FullPath);
            string wctPath = e.FullPath;
            string wctName = e.Name;
            string createdFile = Path.GetFileName(wctName);
            string destPath = @"C:\Users\SomeOtherFolder";
            string sourceFile = wctPath;
            string destFile = Path.Combine(destPath, createdFile);
            WaitForFile(file);
            File.Copy(sourceFile, destFile, true);
        }

        public static bool IsFileLocked(FileInfo file)
        {
            try
            {
                using (FileStream stream = file.Open(FileMode.Open, FileAccess.Read, FileShare.None))
                {
                    stream.Close();
                }
            }
            catch (IOException)
            {
                //the file is unavailable because it is:
                //still being written to
                //or being processed by another thread
                //or does not exist (has already been processed)
                return true;
            }

            //file is not locked
            return false;
        }

        public static void WaitForFile(FileInfo filename)
        {
            //This will lock the execution until the file is ready
            //TODO: Add some logic to make it async and cancelable
            while (!IsFileLocked(filename)) { }
        }

    }
}

What I'm attempting to do in the OnCreated method is to check and wait until the file is done being created, and then copy the file to another destination. I don't seem to know what I'm doing with the WaitForFile(file) line - if I comment out that line and the file creation is instant, the file copies as intended. If I use the WaitForFile line, nothing ever happens. I took the IsFileLocked and WaitForFile methods from other posts on SO, but I'm clearly not implementing them correctly.

I've noted this Powershell version Copy File On Creation (once complete) and I'm not sure if the answer here could be pointing me in the right direction b/c I'm even less versed in PS than I am in C#.

EDIT #1: I should have tested for longer before accepting the answer - I think we're close but after about a minute of the program running, I got the following error before the program crashed:

Unhandled exception. System.IO.IOException: The process cannot access the file 'C:\Users\me\Dropbox est1.log' because it is being used by another process. at System.IO.FileSystem.CopyFile(String sourceFullPath, String destFullPath, Boolean overwrite) at Folderwatch.Program.OnCreated(Object source, FileSystemEventArgs e) in C:\Users\me\OneDrive - Development\Source epos\FolderWatchCG\FolderWatchCG\Program.cs:line 61 at System.Threading.Tasks.Task.<>c.b__139_1(Object state) at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

Any advice on this would be appreciated. As I further analyze the files in these folders, some of them are log files getting written in realtime, so it could be that the file is being written to for hours before it's actually completed. I am wondering if somehow one of the NotifyFilter comes into play here?

user_8967612 · Answer

There's a bug in the WaitForFile() method, that is, it currently waits while the file is not locked (not the other way around). In addition to that, you need a way to confirm that the file actually exists. A simple way to achieve that would be to change the WaitForFile() method into something like this:

public static bool WaitForFile(FileInfo file)
{
    while (IsFileLocked(file))
    {
        // The file is inaccessible. Let's check if it exists.
        if (!file.Exists) return false;
    }

    // The file is accessible now.
    return true;
}

This will keep waiting as long as the file exists and is inaccessible.

Then, you can use it as follows:

bool fileAvailable = WaitForFile(file);
if (fileAvailable)
{
    File.Copy(sourceFile, destFile, true);
}

The problem with this approach though is that the while loop keeps the thread busy, which a) consumes a considerable amount of the CPU resources, and b) prevents the program from processing other files until it finishes waiting for that one file. So, it's probably better to use an asynchronous wait between each check.

Change the WaitForFile method to:

public static async Task WaitForFile(FileInfo file)
{
    while (IsFileLocked(file))
    {
        // The file is inaccessible. Let's check if it exists.
        if (!file.Exists) return false;
        await Task.Delay(100);
    }

    // The file is accessible now.
    return true;
}

Then, await it inside OnCreated like this:

private async static void OnCreated(object source, FileSystemEventArgs e)
{
    // ...

    bool fileAvailable = await WaitForFile(file);
    if (fileAvailable)
    {
        File.Copy(sourceFile, destFile, true);
    }
}

Monitor folder and copy new files once their creation is complete

1 Answers