Powershell filtering one list out of another list

user_9776353 0
Viewed 130

<updated, added Santiago Squarzon suggest information>

I have two lists, I pull them from csv but there is only one column in each of the two lists.
Here is how I pull in the lists in my script

$orginal_list = Get-Content -Path .\random-word-350k-wo-quotes.txt
$filter_words = Get-Content -Path .\no_go_words.txt

However, I will use a typed list for simplicity in the code example below.

In this example, the $original_list can have some words repeated. I want to filter out all of the words in $original_list that are in the $filter_words list.

Then add the filtered list to the variable $filtered_list.
In this example, $filtered_list would only have "dirt","turtle" in it.
I know the line I have below where I subtract the two won't work, it's there as a placeholder as I don't know what to use to get the result.

Of note, the csv file that feeds $original_list could have 300,000 or more rows, and $filter_words could have hundreds of rows. So would want this to be as efficient as possible.
The filtering is case insensitive.

$orginal_list = "yellow","blue","yellow","dirt","blue","yellow","turtle","dirt"
$filter_words = "yellow","blue","green","harsh"

$filtered_list = $orginal_list - $filter_words

$filtered_list

dirt
turtle
filter powershell
2 Answers

Use System.Collections.Generic.HashSet`1 and its .ExceptWith() method:

# Note: if possible, declare the lists as [string[]] arrays to begin with.
#       Otherwise, use a [string[]] cast im the method calls below, which,
#       however, creates a duplicate array on the fly.
[string[]] $orginal_list = "yellow","blue","yellow","dirt","blue","yellow","turtle","dirt"
[string[]] $filter_words = "yellow","blue","green","harsh"

# Create a hash set based on the strings in $orginal_list,
# with case-insensitive lookups.
$hsOrig = [System.Collections.Generic.HashSet[string]]::new(
  $orginal_list,
  [System.StringComparer]::CurrentCultureIgnoreCase
)

# Reduce it to those strings not present in $filter_words, in-place.
$hsOrig.ExceptWith($filter_words)

# Convert the filtered hash set to an array.
[string[]] $filtered_list = [string[]]::new($hsOrig.Count)
$hsOrig.CopyTo($filtered_list)

# Output the result
$filtered_list

The above yields:

dirt
turtle

To also speed up reading your input files, use the following:

# Note: System.IO.File]::ReadAllLines() returns a [string[]] instance.
$orginal_list = [System.IO.File]::ReadAllLines((Convert-Path .\random-word-350k-wo-quotes.txt))
$filter_words = [System.IO.File]::ReadAllLines((Convert-Path .\no_go_words.txt))

Note:

  • .NET generally defaults to (BOM-less) UTF-8; pass a [System.Text.Encoding] instance as a second argument, if needed.

  • .NET's working dir. usually differs from PowerShell's, so the use of full paths is always advisable in .NET API calls, and that is what the Convert-Path calls ensure.

user_453750

I have found that using Linq to filter one list out from another is incredibly easy and incredibly fast (especially for large lists)

# ARRAY OF 1000 STRINGS LOWERCASE (item1 - item1000)
[string[]]$ThousandItems = 1..1000 | %{"item$_"};

# ARRAY OF 100 STRINGS UPPERCASE (ITEM901 - ITEM1000)
[string[]]$HundredItems = 901..1000 | %{"ITEM$_"};

# SUBTRACT THE SECOND ARRAY FROM THE FIRST ONE (CASE INSENSITIVELY)
[string[]]$NineHundred = [Linq.Enumerable]::Except($ThousandItems, $HundredItems, [System.StringComparer]::OrdinalIgnoreCase);
$NineHundred;

Which returns the list of 1000 items minus Item901-Item1000

item1
item2
...
item899
item900

As for speed, removing 100 items from a list...

     1,000 Items =     1ms
    10,000 Items =     2ms
   100,000 Items =    12ms
 1,000,000 Items =   259ms
10,000,000 Items = 3,008ms

Note: These times are just on the [Linq.Enumerable]::Except() line. So it's just measuring the time taken to subtract one array from the other. It does not measure the time taken to fill the array.

So to apply this to the original poster's example

$original_list = [System.IO.File]::ReadAllLines((Convert-Path .\random-word-350k-wo-quotes.txt));
$filter_words = [System.IO.File]::ReadAllLines((Convert-Path .\no_go_words.txt));
[string[]]$filtered_list = [Linq.Enumerable]::Except($original_list,$filter_words,[System.StringComparer]::OrdinalIgnoreCase);

For this, I literally inserted 350K strings (the MD5 hash of the numbers 1 - 350K) into the original list (uppercase), inserted 10K strings (the MD5 hash of the numbers 1-10K) into the filter words list (lowercase) and ran that code.

There were 340K words in the filtered list, and it only took 260ms to read both files, filter and return the list

user_151551900
Related