Slowness to Remove 3,7 and 9 column from | separated txt file using PowerShell

Viewed 86

I have Pipe separated data file with huge data and i want to remove 3,7, and 9 column. below script is working 100% fine. but its too slow its taking 5 mins for 22MB file.

Adeel|01|test|1234589|date|amount|00|123345678890|test|all|01| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|05|test|1234589|date|amount|00|123345678890|test|all|05| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|09|test|1234589|date|amount|00|123345678890|test|all|09| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|00|test|1234589|date|amount|00|123345678890|test|all|00| Adeel|12|test|1234589|date|amount|00|123345678890|test|all|12|

    param
(
    # Input data file
    [string]$Path = 'O:\Temp\test.txt',
    # Columns to be removed, any order, dupes are allowed
    [int[]]$Remove = (3,6)
)

# sort indexes descending and remove dupes
$Remove = $Remove | Sort-Object -Unique -Descending

# read input lines
Get-Content $Path | .{process{
    # split and add to ArrayList which allows to remove items
    $list = [Collections.ArrayList]($_ -split '\|')

    # remove data at the indexes (from tail to head due to descending order)
    foreach($i in $Remove) {
        $list.RemoveAt($i)
    }

    # join and output
    #$list -join '|'
    $contentUpdate=$list -join '|'
    Add-Content "O:\Temp\testoutput.txt" $contentUpdate
}
}
2 Answers

Get-Content is comparatively slow. Use of the pipeline adds additional overhead.

When performance matters, StreamReader and StreamWriter can be a better choice:

param (
    # Input data file
    [string] $InputPath = 'input.txt',
    # Output data file
    [string] $OutputPath = 'output.txt',
    # Columns to be removed, any order, dupes are allowed
    [int[]] $Remove = (1, 2, 2),
    # Column separator
    [string] $Separator = '|',
    # Input file encoding
    [Text.Encoding] $Encoding = [Text.Encoding]::Default
)

$ErrorActionPreference = 'Stop'

# Gets rid of dupes and provides fast lookup ability
$removeSet = [Collections.Generic.HashSet[int]] $Remove

$reader = $writer = $null

try {
    $reader = [IO.StreamReader]::new(( Convert-Path -LiteralPath $InputPath ), $encoding )

    $null = New-Item $OutputPath -ItemType File -Force  # as Convert-Path requires existing path

    while( $line = $reader.ReadLine() ) {

        if( -not $writer ) {
            # Construct writer only after first line has been read, so $reader.CurrentEncoding is available 
            $writer = [IO.StreamWriter]::new(( Convert-Path -LiteralPath $OutputPath ), $false, $reader.CurrentEncoding )
        }

        $columns = $line.Split( $separator )
        $isAppend = $false

        for( $i = 0; $i -lt $columns.Length; $i++ ) {
            if( -not $removeSet.Contains( $i ) ) {
                if( $isAppend ) { $writer.Write( $separator ) }
                $writer.Write( $columns[ $i ] )
                $isAppend = $true
            }
        }

        $writer.WriteLine()  # Write (CR)LF
    }
}
finally {
    # Make sure to dispose the reader and writer so files get closed.
    if( $writer ) { $writer.Dispose() }
    if( $reader ) { $reader.Dispose() }
}
  • Convert-Path is used because .NET has a different current directory than PowerShell, so it's best practice to pass absolute paths to .NET API.
  • If this still isn't fast enough, consider writing this in C# instead. Especially with such "low level" code, C# tends to be faster. You may embed C# code in PowerShell using Add-Type -TypeDefinition $csCode.
  • As another optimization, instead of using String.Split() which creates more sub strings than actually needed, you may use String.IndexOf() and String.Substring() to only extract the necessary columns.
  • Last not least, you may experiment with StreamReader and StreamWriter constructors that lets you allow to specify a buffer size.

Just a more native PowerShell solution/syntax:

Import-Csv .\Test.txt -Delimiter "|" -Header @(1..12) |
Select-Object -ExcludeProperty $Remove |
ConvertTo-Csv -Delimiter "|" -UseQuotes Never |
Select-Object -Skip 1 |
Set-Content -Path .\testoutput.txt

⚠ Note

Many of the techniques described here are not idiomatic PowerShell and may reduce the readability of a PowerShell script. Script authors are advised to use idiomatic PowerShell unless performance dictates otherwise.

  • (Correctly) using the PowerShell Pipeline might save a lot of memory (as every item will be immediately processed and released from memory at the end of the stream -when e.g. sent to disk-) where .Net solutions generally require to load everything into memory. Meaning, at the moment your PC runs out of physical memory and memory pages are swapped to disk, PowerShell might even outperform .Net solutions.
    • As the helpful comment "note that calling Add-Content in every iteration is slow, because the file has to be opened and closed every time. Instead, add another pipeline segment with a single Set-Content call" from @mklement0 implies: the Set-Content cmdlet should be at the end of the pipeline, after the (last) pipe (|) character.
  • The syntax ... .{process{ ... is probably an attempt to Speeding Up the Pipeline. This might indeed improve the PowerShell performance but if you want to implement this properly, you probably don't want to dot-source this but invoke this via the background operator &, See: #8911 Performance problem: (implicitly) dot-sourced code is slowed down by variable lookups.
    Anyways, the bottleneck is likely the input/output device (disk), as long as PowerShell is able to keep up with this input/output device, there is probably no performance improvement in tweaking this.
  • Besides the fact that the later PowerShell versions (7.2) generally perform better than Windows PowerShell (5.1), some cmdlets are improved. Like the newer ConvertTo-Csv which has additional -UseQuotes <QuoteKind> and -QuoteFields <String[]> parameters. If you are stuck with Windows PowerShell, you might check this question: Delete duplicate lines from text file based on column
  • Although there is an easy way to read delimited files without headers using the Import-Csv cmdlet (with the -header parameter) using the Import-Csv cmdlet, there is no easier way to skip the Cvs header for the counter cmdlet Export-Csv. This can be worked around with the with: ConvertTo-Csv |Select-Object -Skip 1 |Set-Content -Path .\output.txt, see also: #17527 Add -NoHeader switch to Export-Csv and ConvertTo-Csv
Related