Monitoring file system changes and logging to file

Question

My actual goal is to monitor a folder for files being created and in which order they are created. Maybe there is a better tool than powershell. If so please let me know.

I took and tailored the second script here to monitor file system changes.

It uses System.IO.FileSystemWatcher to monitor the file system and fire events for certain changes in a folder.

I need to write the result to a file. First try was to pipe to Out-File. As we have tons of changes each and every second performance was very bad.

Then I took StreamWriter and and WriteLine'd the events. Performance is better know but still not perfect.

Would it make sense to use the asnchronous version of WriteLine? Or is there a even better way?


# specify the path to the folder you want to monitor:
$Path = "D:\Transfer\"

# specify whether you want to monitor subfolders as well:
$IncludeSubfolders = $true

# specify the file or folder properties you want to monitor:
$AttributeFilter = [IO.NotifyFilters]::FileName

try
{
  $watcher = New-Object -TypeName System.IO.FileSystemWatcher -Property @{
    Path = $Path
    IncludeSubdirectories = $IncludeSubfolders
    NotifyFilter = $AttributeFilter
  }

  # define the code that should execute when a change occurs:
  $action = {
    # the code is receiving this to work with:
    
    # change type information:
    $details = $event.SourceEventArgs
    $FullPath = $details.FullPath
    $OldFullPath = $details.OldFullPath
    
    # type of change:
    $ChangeType = $details.ChangeType
    
    # when the change occured:
    $Timestamp = ($event.TimeGenerated).ToString("yyyyMMddHHmmssffffZ")
    $TimestampNow = (Get-Date).ToString("yyyyMMddHHmmssffffZ")
    
    # now you can define some action to take based on the
    # details about the change event:
    
    # you can also execute code based on change type here:
    switch ($ChangeType)
    {
      "Created"  { $text = "{0}`t{1}`t{2}`t{3}" -f $TimestampNow, $Timestamp, $ChangeType, $FullPath }
      "Deleted"  { $text = "{0}`t{1}`t{2}`t{3}" -f $TimestampNow, $Timestamp, $ChangeType, $FullPath }
      "Renamed"  { $text = "{0}`t{1}`t{2}`t{3}`t{4}" -f $TimestampNow, $Timestamp, $ChangeType, $FullPath, $OldFullPath }
      "Error"  { $text = "{0}`t`t{1}" -f $TimestampNow, $ChangeType }
    }

    $global:SW.WriteLine($text)
    $global:SW.Flush()
  }

  # subscribe your event handler to all event types that are
  # important to you. Do this as a scriptblock so all returned
  # event handlers can be easily stored in $handlers:
  $handlers = . {
    Register-ObjectEvent -InputObject $watcher -EventName Created  -Action $action 
    Register-ObjectEvent -InputObject $watcher -EventName Deleted  -Action $action 
    Register-ObjectEvent -InputObject $watcher -EventName Renamed  -Action $action 
  }

  # monitoring starts now:
  $watcher.EnableRaisingEvents = $true

  Write-Warning "Watching for changes to $Path"

  # since the FileSystemWatcher is no longer blocking PowerShell
  # we need a way to pause PowerShell while being responsive to
  # incoming events. Use an endless loop to keep PowerShell busy:
  do
  {
    # Wait-Event waits for a second and stays responsive to events
    # Start-Sleep in contrast would NOT work and ignore incoming events
    Wait-Event -Timeout 1
  } while ($true)
}
finally
{
  # this gets executed when user presses CTRL+C:
  
  # stop monitoring
  $watcher.EnableRaisingEvents = $false
  
  # remove the event handlers
  $handlers | ForEach-Object {
    Unregister-Event -SourceIdentifier $_.Name
  }
  
  # event handlers are technically implemented as a special kind
  # of background job, so remove the jobs now:
  $handlers | Remove-Job
  
  # properly dispose the FileSystemWatcher:
  $watcher.Dispose()
  
  $global:SW.Close()

  Write-Warning "Event Handler disabled, monitoring ends."
}

user_45375 · Answer

I suggest calling Register-ObjectEvent without an -Action argument, and instead letting PowerShell queue the events internally.
You can then process each event in the queue sequentially with the Wait-Event cmdlet, directly in a loop.

I'm assuming that this improves performance, because there is the no need to invoke a script block hosted in a dynamic module that must be called for each event (which is what -Action does).

General caveats:

The .NET System.IO.FileSystemWatcher API itself can miss events, because it is notified of events via a buffer of fixed size; the buffer size can be increased, however. See the Remarks section for more information.
On the PowerShell side, I don't think there's a queue size limit, so you can at least hypothetically run out of memory if you don't consume the queue fast enough.
Events do seem to be reported by PowerShell in chronological order, but note the following:
- What is a single file operation from the user's perspective may result in multiple events, notably when files are moved - again, see the "Remarks" link above.
  - Update: You report that the two events that are reported in your case for a move - Created and Deleted - are unexpectedly out of chronological order: sensibly, creating the file in the new location happens before deleting it from the old location, as reflected in their event timestamps, but you received the Deleted event first.
- If you use wildcards to delete files (e.g. Remove-Item *), the files aren't necessarily deleted in alphabetical order, so the order of deletion events doesn't reflect alphabetical order either.

The following adaptation of your code demonstrates this technique. Note that what is written to the log file for each event is simplified in the interest of brevity:

$Path = "/Users/mklement/Desktop/pg/tf"
$logFile = '/Users/mklement/Desktop/pg/o.txt' # Be sure to use a FULL PATH.

# specify whether you want to monitor subfolders as well:
$IncludeSubfolders = $true

# specify the file or folder properties you want to monitor:
$AttributeFilter = [IO.NotifyFilters]::FileName

try
{
  $watcher = [System.IO.FileSystemWatcher] @{
    Path = $Path
    IncludeSubdirectories = $IncludeSubfolders
    NotifyFilter = $AttributeFilter
  }

  # Subscribe your event handler to all event types that are
  # important to you.
  # Note: 
  #  * The event names also serve as the -SourceIdentifier values, for
  #    later unregistering with Unregister-Event.
  #  * Without -Action, Register-ObjectEvent produces NO output.
  $eventNames = 'Created', 'Deleted', 'Renamed'
  $eventNames | ForEach-Object {
    Register-ObjectEvent -InputObject $watcher -EventName $_ -SourceIdentifier $_
  }

  # monitoring starts now:
  $watcher.EnableRaisingEvents = $true

  Write-Warning "Watching for changes to $Path"

  # Create the stream writer for the log file.
  $sw = [System.IO.StreamWriter] $logFile

  # Process events one by one, as they arrive, indefinitely.
  # Ctrl-C is needed to exit.
  while ($true) {
    # Synchronously wait for the next event / process the next queued one.
    # Note: *Any* event is waited for here, so the assumption is that
    #       your session is only subscribed to file system-watcher events.
    #       If necessary, you could filter out unrelated ones.
    $evt = Wait-Event
    $evt | Remove-Event # Events must be removed manually from the queue.
    # Write the event details to the log file...
    $sw.WriteLine(
      $evt.SourceEventArgs.psobject.Properties.Value -join "`t"
    )
    # ... and flush it asynchronously.
    $sw.FlushAsync()
  }

}
finally
{
  # This gets executed when user presses CTRL+C:

  # Close the log file.
  $sw.Close()

  # Unregister the event subscriptions.
  $eventNames | ForEach-Object { Unregister-Event -SourceIdentifier $_ }

  # properly dispose of the FileSystemWatcher:
  $watcher.Dispose()
  
  Write-Warning "Event Handler disabled, monitoring ends."
}

Monitoring file system changes and logging to file

1 Answers