Powershell Compare two directories files (Name and Hash). Report if the name is only in one dir but ignore it for the hash check

This is how I would do it, it's more code than what you already have but I believe should do what you're looking for.

The firsts steps are mainly the same, gather all SHA1 hashes from both folders and construct a list of objects.
Then, on the first loop over the "difference folder" (this would be the Copy folder), filter by Unique files based on it's hashes compared to the reference folder and files where the Name of the file exists in the reference folder but it's hash has changed (this would meet the condition 2 of your question).
Lastly, the second loop over the "reference folder" to get the unique files of this collection based on the file hashes.

The $result collection would be an array of objects with a newly added property Status to help identify if the file has changed or if it's unique.

$result = [System.Collections.Generic.List[object]]::new()

$sb = {
    process {
        if($_.Name -eq 'Thumbs.db') { return }

        [PSCustomObject]@{
            h  = (Get-FileHash $_.FullName -Algorithm SHA1).Hash
            n  = $_.Name
            s  = $_.Length
            fn = $_.fullname
        }
    }
}

$refFiles  = Get-ChildItem 'path/to/reference/folder' -Recurse -File | & $sb
$diffFiles = Get-ChildItem 'path/to/difference/folder' -Recurse -File | & $sb

foreach($file in $diffFiles) {
    # this file exists on both folders, skip it
    if($file.h -in $refFiles.h) { continue }
    # this file exists on reference folder but has changed
    if($file.n -in $refFiles.n) {
        $file.PSObject.Properties.Add(
            [psnoteproperty]::new('Status', 'Changed in Ref')
        )
        $result.Add($file)
        continue
    }
    # this file does not exist on reference folder
    # based on previous conditions
    $file.PSObject.Properties.Add(
        [psnoteproperty]::new('Status', 'Unique in Diff')
    )
    $result.Add($file)
}

foreach($file in $refFiles) {
    # this file is unique in reference folder, rest of the files
    # not meeting this condition can be ignored since we're
    # interested only in files on reference folder that are unique
    if($file.h -notin $diffFiles.h) {
        $file.PSObject.Properties.Add(
            [psnoteproperty]::new('Status', 'Unique in Ref')
        )
        $result.Add($file)
    }
}

$result | Format-Table

I was playing with Compare-object and I was able to get it to work by creating a sub array. Thanks to a post by @Theo. The answer from @Santiago Squarzon works, but I was still curious if I could do it.

$VDrive = "C:\Temp"
$Set = "GroupA"
function check-hash{
$DefaultDir = "$VDrive\Default\Files"
$SetDir = "$VDrive\$Set\Files"
    
    $SetArray = @()
    $SetArray = @((Get-ChildItem -File -Path $SetDir -Recurse) | Where-Object { $_ -notmatch "Thumbs.db"}| 
    ForEach-Object {
        [PSCustomObject]@{
        h = (Get-FileHash $_.FullName -Algorithm SHA1).Hash
        n = $_.Name
        pn = $($_.fullname).Replace("C:\Temp\$Set", "")
        fn = $_.fullname
        }})
   
    $DefaultArray = @()
    $DefaultArray = @((Get-ChildItem -File -Path $DefaultDir -Recurse) | Where-Object { $_-notmatch "Thumbs.db"}| 
    ForEach-Object {
        [PSCustomObject]@{
        h = (Get-FileHash $_.FullName -Algorithm SHA1).Hash
        n = $_.Name
        pn = $($_.fullname).Replace("C:\Temp\Default", "")
        fn = $_.fullname
        }})

    Write-host "Comparing..."
    $compare = Compare-Object -ReferenceObject $DefaultArray  -DifferenceObject $SetArray -Property h,pn -IncludeEqual -PassThru | Select-Object n,pn,SideIndicator,fn
    $filecomp = Compare-Object -ReferenceObject $DefaultArray  -DifferenceObject $SetArray -Property pn -IncludeEqual -PassThru | Select-Object n,pn,SideIndicator,fn
        $doSet = $compare | Where-Object { $_.SideIndicator -eq '<=' }
        $doDefault = $compare | Where-Object { $_.SideIndicator -eq '=>' }
        $foSet = $filecomp | Where-Object { $_.SideIndicator -eq '<=' }
        $foDefault = $filecomp | Where-Object { $_.SideIndicator -eq '=>' } 

        if ($foSet) {
            Write-Host "File Delta from Default"  -ForegroundColor Yellow
            $foSet | Format-Table -AutoSize
            }
    
        if ($foDefault) {
            Write-Host "File Delta From Set"  -ForegroundColor Green
            $foDefault | Format-Table -AutoSize
            }
               
#####   Try comparing doSet and doDefault ############################################
    $Dcomp = Compare-Object -ReferenceObject $doSet  -DifferenceObject $doDefault -Property pn -IncludeEqual -PassThru | Select-Object n,pn,SideIndicator,fn
        $DcompSwim = $Dcomp | Where-Object { $_.SideIndicator -eq '==' }
    
        if ($DcompSwim) {
            Write-Host "Hash Delta from Default"  -ForegroundColor Cyan
            $DcompSwim | Format-Table -AutoSize
            }
    
}
check-hash