Memory leak when searching multiple log file patterns with Powershell

user_16933196 0
Viewed 106

I am writing a script that searches for a specific pattern in multiple log files with Powell Shell. There is a serious memory leak when executing the script, so I am wondering how to search the files one by one. Help.

$LOG_FILE = 'C:\test\test\test_*.log
$FIND_LINE = Get-Content $LOG_FILE | findstr 'test'
powershell
2 Answers

Get-Content is slow and memory-hungry, so it's best to let Select-String - PowerShell's more fully-featured analog to findstr.exe - search the files of interest directly:

$linesFound = (Select-String -Path C:\test\test\test_*.log -Pattern 'test').Line

Note:

  • By default, the -Pattern argument is interpreted as a regex; add the -SimpleMatch switch to perform a verbatim substring search.

  • Select-String, like PowerShell in general, is case-insensitive by default;
    add -CaseSensitive if needed.

  • Select-String wraps the lines it finds in [Microsoft.PowerShell.Commands.MatchInfo] instances that contain metadata about each match; accessing the .Line property returns the matching line.

user_453750

Get the log files into an array and then loop through them. Don't use findstr as that is not native powershell its an old cmd executable. Use Select-String as others have said.

[array]$LOG_FILE = Get-ChildItem C:\test\test\test_*.log
[string[]]$FIND_LINE = ForEach ($L in $LOG_FILE) { Get-Content $L | Select-String 'test' }

Then you could export the results to a text file like so:

$FIND_LINE | Out-File "C:\test\FoundLines.txt"

Since you mentioned memory leaks and such. Powershell can eat memory it never returns within a given session but if you save this into a script and execute it as a script then it should clear up memory after the script ends. However there are a few tricks to clean up memory within loops like this dotnet command for garbage collection:

[system.gc]::Collect()

It will add a tiny bit of processing time to each iteration of the loop though so I would test how long it takes with and without the garbage collection to make sure it's worth it. I suggest also reading about some other methods and caveats with that mentioned here: http://www.jhouseconsulting.com/2017/09/25/addressing-the-powershell-garbage-collection-bug-1825

And also use PowerShell 7 (if you can install it) as it would likely be faster and more efficient.

user_72031040
Related