Powershell move duplicates to a different folder

user_14457814 0
Viewed 317

I need to spawn through a directory to find duplicates using PowerShell. Skip the first one in the list of duplicate files and move the rest to another folder. I got the below script for a start, but could you help me with how to implement the move rest of the files.

Trap { continue }
$Date = Get-Date
Get-ChildItem *.* -Recurse | 
Where-Object { $_.LastAccessTime -lt $Date.AddDays(-2555) } | 
Get-FileHash | 
Group-Object -Property Hash | 
Where-Object Count -gt 1 | 
ForEach-Object { $_.group | Select-Object -skip 1 } | 
<move the files>
powershell
1 Answers

Given comments on the original question the exact requirements aren't clear. Name collisions are only possible if the same name is encountered in different sub-folders. Therefore, all the files with the same hash have the same data and "possibly" the same name. Assuming there's a real potential for name collisions AND a need to copy all but the first of the files in a given hash group the key piece would look something like:

... |
  ForEach-Object { 
    $_.Group | Select-Object -Skip 1 | Move-Item -Force -Destination $otherFolder
  }

It's important to note this will move everything except the first file in the group. i.e. if you have 3 files in the group it will move the last 2. If there's a name collision among those 2, the 2nd will overwrite the 1st and you will only have that file in the destination.

Note: The code may perform somewhat better if you move the Move-Item command after the ForEach-Object loop, like:

... |
  ForEach-Object { 
    $_.Group | Select-Object -Skip 1
  } |
Move-Item -Force -Destination $otherFolder

In this case the [FileInfo] objects will be emitted from the loop piped to and consumed by Move-Item

Now, considering the stated ambivalence to overwrites I suspect the summary requirement is to simply get "a" copy of the data irrespective of file name. Moreover, given there's been no discussion of sorting, or other selection criteria to determine which file to skip, it may be fine to use -Last 1 instead. This might be more efficient in cases where there's more than 2 files in the group. However, it won't move 2nd... files in a group that have different names which is what distinguishes this approach from -Skip 1.

... |
  ForEach-Object { 
    $_.Group | Select-Object -Last 1 | Move-Item -Force -Destination $otherFolder
  }

Thanks to @mklement0, for helping to identify and distinguish between these 2 approaches.

Additional Information:

I also think you should filter out directories. The directory objects will cause Get-Hash to error. Doing so is quite simple:

Get-ChildItem -Recurse -Filter *.* -File |
...

Generally speaking, it's better to avoid errors where you know they can occur and can indeed avoid them.

I suspect the Trap{ continue } statement was to deal with the errors coming from Get-Hash. Notwithstanding the potential need for error handling elsewhere That could have been handled by setting -ErrorAction SilentlyContinue

A Consolidated Example:

$DateBound = (Get-Date).AddDays(-2555)
Get-ChildItem -Recurse -Filter *.* -File | 
Where-Object { $_.LastAccessTime -lt $DateBound } | 
Get-FileHash -ErrorAction SilentlyContinue | 
Group-Object -Property Hash | 
Where-Object { $_.Count -gt 1 } | 
ForEach-Object { 
    $_.Group | 
    Select-Object -Last 1 |
    Move-Item -Force -Destination $otherFolder
}

Note: Above I calculated the date boundary in variable DateBoundary beforehand. This is more efficient because you don't have to recalculate the date for every file encountered.

In general, I'd avoid Trap{} it's antiquated, and the favored approach is the Try{} Catch{} Finally{}, Recommended reading: about_Try_Catch_Finally.

One last thing; You were mixing simplified syntax with traditional syntax. I never use simplified and I don't know if there's a community opinion on it. However, at a minimum, I would advise staying consistent within a given unit of code.

user_47492640
Related