Powershell's memory limits?

irvnriir 0
Viewed 118

why does this not reach $StringBuilder.MaxCapacity (either 2GB) in Windows's Powershell ?

$StringBuilder = [System.Text.StringBuilder]::new()
$nu_s = 0
$nu = 100000000
while ( $null -ne $StringBuilder.Append([char[]]::new($nu)).ToString() ) {
    $nu_s = $nu_s + $nu
}
while ($nu -gt 1) {
    # $null = $StringBuilder.Remove($nu_s, $StringBuilder.Length - $nu_s)
    $nu = $nu / 10
    while ( $null -ne $StringBuilder.Append([char[]]::new($nu)).ToString() ) {
        $nu_s = $nu_s + $nu
    }
}
Write-Host $nu_s, $StringBuilder.MaxCapacity
$StringBuilder.Clear()

i suppose this is related to some memory limits in P, does anyone know which that is and how to fix ?

edit_1: ~~also, the StringBuilder's Constructor doesn't accept values higher than the script provide .~~ ( it does, but still less than $StringBuilder.MaxCapacity )

edit_2: i tried using [GC]::Collect(), [GC]::WaitForPendingFinalizers() and $v.Finalize(), no changes .

edit_3: im giving up, the s**t P refuses to allow to fix this . im going with all-cases string and array limits: 200 000 000 Bytes/items; and preferable: 100 000 000 Bytes/items . (i had this issue with array-related scripts too)

powershell .net
1 Answers

edit_final

  • <<StringBuilder>>.Append crashes before reaching <<StringBuilder>>.MaxCapacity , because it's content is not stored with the most efficient layout, use 5823-items [char[]] -s to store the maximum of items .

    also, notice that maximum of items is specified by that <<StringBuilder>>.Length is limited to the [int] 's max . [pointed out by @phuclv]

  • the issue when i can't use more than 2GB of memory, is inherited from the environment . it can be determined by:

    • in case its because the Powershell runs in 32-bit -- [Environment]::Is64BitProcess . [pointed out by @phuclv]
    • as a workaround, loading the memory until limit is reached, using analog of the script below .

  • also, [char] is a 2-bytes UTF-16 item, [System.Array] is limited to about 2GB (a bit more) .

edit: also see the comments below .

i've read that Windows ~processes by default get a limit of 2GB of Primary Storage . (it would limit the StringBuilder's content)

this script should get the exact value which can be used as a limit .

  • i get different max values in different environments . but these are: about 2GB and about 4GB, and as i've read, the OS's limit can be increased to 3GB and 4GB, so i assume that this limit is the reason of the issue .
  • PrivateMemorySize64 results differ about 2KB (and each the other, except SystemMemory, differ too), so the limit still needs a margin, that makes it not false-proof, but unless someone posts how to get Powershell's view of it's resources, this is the workaround provided .
$ErrorActionPreference = 'SilentlyContinue'
$StringBuilder = [System.Text.StringBuilder]::new()
$Process = [System.Diagnostics.Process]::GetCurrentProcess()
$showMemoryLoad = {
    Write-Host ''
    $args[0].Refresh()
    Write-Host "PagedMemorySize64               = $($args[0].PagedMemorySize64)"        # prob equal to PrivateMemorySize64
    Write-Host "PrivateMemorySize64             = $($args[0].PrivateMemorySize64)"
    Write-Host "PagedSystemMemorySize64         = $($args[0].PagedSystemMemorySize64)"
    Write-Host "physical memory currently used  = $($args[0].WorkingSet64)"
    Write-Host "NonpagedSystemMemorySize64      = $($args[0].NonpagedSystemMemorySize64)"
    Write-Host "VirtualMemorySize64             = $($args[0].VirtualMemorySize64)"
}

$limit = 10000000000
$nu_sc = $limit / 10 * 10
while ($nu_sc -gt 1) { $nu_sc = $nu_sc / 10
    :sc for ($nu_re_sc = 0; $nu_re_sc -lt 9; $nu_re_sc++) {
        $limit = $limit - $nu_sc
        Write-Host ''
        Write-Host $limit
    
        $null = $StringBuilder.Clear()
        $nu = 100000000 * 10
        while ($nu -gt 1) {
            $nu = $nu / 10
            Write-Host ''
            Write-Host "$nu $($Process.PrivateMemorySize64) $($StringBuilder.Length)"
            while (($limit - $Process.PrivateMemorySize64) -gt ($nu * 1)) {
                $v = $StringBuilder.Append([char[]]::new($nu))
                if ($null -eq $v) {continue sc}
                $v = $null
            }
        }
        
        $limit = $limit + $nu_sc
        break
    }
}
& $showMemoryLoad $Process
Write-Host ''
Write-Host $StringBuilder.Length, $StringBuilder.MaxCapacity $limit
irvnriir0
Related