Powershell doesn't read files size correctly if these are in progress of change (i.e files being downloaded)

This answer is meant to demonstrate that, even though unreliable, file size can be monitored in real-time. In this case [System.IO.StreamWriter] is writing to a file adding 1 byte on each iteration until the the file reaches 1Mb.

I have also performed the same test while downloading a "test file" and it's working fine for me, the total size of the folder is being properly updated.

The possible causes for what you're observing and purely based on assumptions:

• The file being download has pre-allocated space - I don't see how that would be possible since, as you explained, by looking on Explorer you can see the file size increasing.
• There is not enough time between the size before vs the size after calculation - This could be due to buffering, mklement0 has explained this in detail. Increasing the sleep time might solve this problem. Also, calling the .Refresh() method which I was not aware of, TIL thanks :)

My thoughts, I personally don't think this is the right approach to solve this problem. Process monitoring would be a much better approach (assuming this is a possibility - also explained by mklement0 and I totally agree with him).

$checkSize = {
    (Get-ChildItem $PWD -File | Measure-Object -Property Length -Sum).Sum
}

$currentSize = & $checkSize
$testFile = Join-Path $pwd -ChildPath "testfile.dump"

$job = Start-Job {

    $writer = [System.IO.StreamWriter]::new(   
        [System.IO.File]::Create($using:testFile)
    )
    0..1Mb | ForEach-Object { $writer.Write(1) }
    $writer.Close()

} -Name testDump

'
Starting test:
'

do
{
    $increasingSizeBefore = & $checkSize
    Start-Sleep -Seconds 2
    $increasingSizeAfter = & $checkSize

    'StartingSize: {0} - SizeBefore: {1} - SizeAfter: {2}' -f
    $currentSize, $increasingSizeBefore, $increasingSizeAfter

} until ($increasingSizeBefore -eq $increasingSizeAfter)

$job | Stop-Job -PassThru | Remove-Job

Results for me:

Starting test:

StartingSize: 17458 - SizeBefore: 17458 - SizeAfter: 152626
StartingSize: 17458 - SizeBefore: 152626 - SizeAfter: 406578
StartingSize: 17458 - SizeBefore: 406578 - SizeAfter: 652338
StartingSize: 17458 - SizeBefore: 652338 - SizeAfter: 902194
StartingSize: 17458 - SizeBefore: 902194 - SizeAfter: 1066035
StartingSize: 17458 - SizeBefore: 1066035 - SizeAfter: 1066035

I suggest adopting a different strategy:

• Set an overall timeout for each download process, such as curl.exe's --max-time option.

• Unfortunately, PowerShell's own Invoke-WebRequest and Invoke-RestMethod appear to have only a connection timeout (-TimeoutSec), not a timeout for the overall connection.

That way you can track the download processes, and trigger a reboot once all of them have terminated (whether due to completion or timeout).

As for your approach:

• The on-disk file size that you can query via Get-ChildItem is not updated continuously while a file is being written, as you've observed, and while it is eventually updated, that may not happen until the file has been closed, i.e. written in full.

  • You can experiment with throttled writes based on Santiago Squarzon's helpful answer.

• However, you can update the file-size information on demand, namely via the System.IO.FileSystemInfo.Refresh() method, which is the equivalent of the manual refreshing you performed via File Explorer.

  • Note, however, that this still isn't real-time size information due to internal buffering of writes.

# Perform this before every Measure-Object call.
# It refreshes the size information of all files in the specified dir.
(Get-ChildItem -File -LiteralPath $downloadDir).Refresh()

As an aside: As Santiago points out, this approach fundamentally wouldn't work for download utilities / APIs that preallocate the output file with the full size of the download, which is a feature that some BitTorrent clients apparently offer.

As for narrowing down what downloads have completed vs. which ones are assumed to be stuck:

Invoke-WebRequest / Invoke-RestMethod exclusively lock their output files while a download is ongoing, so you can make a read attempt to see which files cannot be read from, from which you can infer what downloads, if any, are still ongoing:

# Note: In PowerShell (Core) 7+, use -AsByteStream instead of -Encoding Byte
Get-ChildItem -File -LiteralPath $downloadDir | 
  Get-Content -Encoding Byte -First 1 -ErrorVariable errs -ErrorAction SilentlyContinue |
    Out-Null

if ($errs) { Write-Warning "Incomplete downloads:`n$($errs.TargetObject)" }