Powershell slowness in "get-childitem . -directory -recurse" when there are lots of files

Get-ChildItem is known to be slow to traverse directories recursively but it is what is built-in to PowerShell, it's a very handy and easy to use cmdlet. If you're looking for speed and efficiency you may need to default to .NET API calls to IO.Directory.

Haven't tested this enough but I believe it should be working, as in, finding any directory that is empty and is older than 90 days in this case.

Worth noting that this code requires .NET Framework 4+ if running Windows PowerShell.

$queue = [Collections.Generic.Queue[IO.DirectoryInfo]]::new()
$olderThan = [datetime]::Now.AddDays(-90)         # => Set limit date here!
$queue.Enqueue('F:\logs\PRV_RequestLogs\inbound') # => Starting path here!

while($queue.Count) {
    $dir = $queue.Dequeue()
    try {
        $isEmpty = $true
        foreach ($i in $dir.EnumerateDirectories()) {
            if($i.Attributes -band [IO.FileAttributes]::ReparsePoint) {
                # skip if it is ReparsePoint
                continue
            }
            $isEmpty = $false
            $queue.Enqueue($i)
        }

        if($isEmpty -and $dir.CreationTime -lt $olderThan -and -not $dir.EnumerateFiles().GetEnumerator().MoveNext()) {
            # output the directory if the 3 conditions are `$true`, no folders or files and older than
            $dir
        }
    }
    catch {
        # if we were unable to enumerate this directory, go next
        continue
    }
}

Relevant Documentation

• DirectoryInfo Class
• EnumerateDirectories() instance method
• EnumerateFiles() instance method
• Queue<T> Class
• FileAttributes Enum

To complement Santiago Squarzon's helpful answer:

In addition to the implementation inefficiency you have discovered, there is another reason Get-ChildItem is slow: it decorates each System.IO.FileInfo / System.IO.DirectoryInfo output object with instance-level ETS (Extended Type System) provider properties - see GitHub issue #7501, which explains the problem and suggests a streamlined class-level solution.

Using .NET APIs directly is indeed needed to work around the performance problems - a general caveat is that you should always use full paths when calling .NET methods, because .NET's working directory usually differs from PowerShell's.

The fast equivalent of:

Get-ChildItem F:\logs\PRV_RequestLogs\inbound -Recurse -Directory | % { $_.fullname }

is the following, which uses the [System.IO.Directory]::EnumerateDirectories() method:

[System.IO.Directory]::EnumerateDirectories(
  'F:\logs\PRV_RequestLogs\inbound', 
  '*', 
  'AllDirectories'  # Equivalent of -Recurse, a [System.IO.SearchOption] enum value
)

Caveats:

• Inaccessible directories:

  • In Windows PowerShell / .NET Framework they invariably cause an exception that aborts the enumeration.

  • In PowerShell (Core) 7+ / .NET you now have to option to ignore inaccessible directories, via a new method overload that accepts a [System.IO.EnumerationOptions] instance.

• Symlinks / reparse points:

  • In Windows PowerShell / .NET Framework they are invariably followed (including by Get-ChildItem -Recurse).
  • In PowerShell (Core) 7+ / .NET you now have to option to opt-out (and Get-ChildItem now requires opt-in via -FollowSymlink)

The [System.IO.DirectoryInfo] class too has a .EnumerateDirectories() method, albeit as an instance method; the major difference is that [System.IO.DirectoryInfo] instances rather than path strings are returned:

([System.IO.DirectoryInfo] 'F:\logs\PRV_RequestLogs\inbound').EnumerateDirectories(
  '*', 
  'AllDirectories'
)

You can leverage the above to perform the desired filtering - finding only empty directories that are older than N days - with a minimum of PowerShell code:

$cutOffDate = (Get-Date).Date.AddDays(-30) # adjust as needed.
([System.IO.DirectoryInfo] 'F:\logs\PRV_RequestLogs\inbound').EnumerateDirectories(
  '*', 
  'AllDirectories'
) | Where-Object { 
      $_.LastWriteTime -lt $cutOffDate -and  
      0 -eq ($_.EnumerateFileSystemInfos() | Select-Object -First 1).Count
    }

Note the use of the .EnumerateFileSystemInfos() method to enumerate all files and subdirectories, followed by selecting - at most - one via Select-Object -First 1, which stops the enumeration as soon as one item has been enumerated. A .Count value of 0 implies that the directory at hand is empty.