Remove Redundance via bash

Doing this properly requires a lot of internal sorting which inside a bash loop spawns a lot of processes and slows the job down too much.

Switching to perl. Note that this reorders arguments as well as lines; if you need the original lines untouched and/or in the original order, we'll have to add another step or three. You should also note that you have knowledge both capitalized and lowercase; url's are case insensitive through the port, but path after that is case sensitive, so they will not register as the same even if they get identical arguments.

#!/usr/bin/env perl

use strict;     # I ALWAYS use strict and warnings unless 
use warnings;   # there is some compelling reason not to.

open my $fh, 'urls' or die "urls: $!";
my %urlsOUT;
foreach ( <$fh> ) { chomp;
    my %args;                              # clean for each record
    m!^(https?://[^/]+)(/[^?]+)[?](.*)!i;  # catch the base in separate case sensitivities
    my ($base) = lc($1).$2;                # always lowercase the case insensitive part
    @args{ split /[?&]+/, $3 } = ();       # removes duplicate args in a url
    my ( $args ) = join '&', reverse sort keys %args; # reassemle ORDERED
    $urlsOUT{"$base?$args"}='';            # now a unique key
}

my $urlsOUT='';
REC: foreach my $url (reverse sort keys %urlsOUT ) { # ORDERED
       for ( split /[?&]/, $url ) {                  # for each arg
         if ( $urlsOUT !~ /\b$_\b/ ) {               # if new
           $urlsOUT .= "$url\n";                     # keep this
           next REC;                                 # check next
         }
       }
}

print $urlsOUT;

This will consistently reorder and de-dup all arguments in a URL, de-dup all resulting records, and then check each remaining record (in descending order) to eliminate any record that doesn't have something no other record before it had.

I named the program file tst and made a tst1 and a urls.

$: cat tst1
http://test/foo?foo
http://test/foo?bar
http://test/foo?foo
http://test2/foo?foo
http://test2/foo?baz
http://test2/foo?foo&bar
http://test2/foo?baz
http://test/foo?foo&bar
http://test/foo?bar&foo
http://test2/foo?bar&foo
http://test3/foo?bar
http://test3/foo?foo&bar&baz
http://test2/foo?foo&bar&baz
http://test/foo?foo&bar&baz

$: ./tst tst1
http://test3/foo?foo&baz&bar
http://test2/foo?foo&baz&bar
http://test/foo?foo&baz&bar

$: cat urls
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&docID=111
http://grouplogic.com:80/news-events/index.cfm?fa=viewNews&ID=390
http://grouplogic.com:80/public/quickpoll/index.cfm?fuseaction=quickPollResults&QuestionID=8
http://grouplogic.com:80/store/index.cfm?cfid=11812682&cftoken=26157811&fa=conre
http://grouplogic.com:80/store/index.cfm?fa=PrtSlt&id=532&prTpID=5&
http://grouplogic.com:80/store/index.cfm?upTp=2&fa=upgrade&UpNewType=2&prTpID=5&&ptype=FS
http://grouplogic.com:80/news-events/index.cfm?fa=viewRelease&ID=21&prod=2
http://grouplogic.com:80/content/index.cfm?fuseaction=faq_list&ProdID=1&archive=1
http://grouplogic.com:80/content/index.cfm?ID=103
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&amp
http://grouplogic.com:80/knowledge/index.cfm?fuseaction=view&docID=10
http://grouplogic.com:80/content/index.cfm?ID=123
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&docID=111
http://grouplogic.com:80/news-events/index.cfm?fa=viewNews&ID=390
http://grouplogic.com:80/public/quickpoll/index.cfm?fuseaction=quickPollResults&QuestionID=8
http://grouplogic.com:80/content/index.cfm?ID=123&foo=bar
http://grouplogic.com:80/store/index.cfm?cfid=11812682&cftoken=26157811&fa=conre
http://grouplogic.com:80/store/index.cfm?fa=PrtSlt&id=532&prTpID=5&
http://grouplogic.com:80/store/index.cfm?    upTp=2&fa=upgrade&UpNewType=2&prTpID=5&&ptype=FS
http://grouplogic.com:80/news-events/index.cfm?fa=viewRelease&ID=21&prod=2
http://grouplogic.com:80/content/index.cfm?fuseaction=faq_list&ProdID=1&archive=1
http://grouplogic.com:80/content/index.cfm?ID=103
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&amp
http://grouplogic.com:80/knowledge/index.cfm?fuseaction=view&docID=10
http://grouplogic.com:80/content/index.cfm?ID=123

$: ./tst urls
http://grouplogic.com:80/store/index.cfm?upTp=2&ptype=FS&prTpID=5&fa=upgrade&UpNewType=2
http://grouplogic.com:80/store/index.cfm?prTpID=5&id=532&fa=PrtSlt
http://grouplogic.com:80/store/index.cfm?fa=conre&cftoken=26157811&cfid=11812682
http://grouplogic.com:80/public/quickpoll/index.cfm?fuseaction=quickPollResults&QuestionID=8
http://grouplogic.com:80/news-events/index.cfm?prod=2&fa=viewRelease&ID=21
http://grouplogic.com:80/news-events/index.cfm?fa=viewNews&ID=390
http://grouplogic.com:80/knowledge/index.cfm?fuseaction=view&docID=10
http://grouplogic.com:80/content/index.cfm?fuseaction=faq_list&archive=1&ProdID=1
http://grouplogic.com:80/content/index.cfm?foo=bar&ID=123
http://grouplogic.com:80/content/index.cfm?ID=103
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&docID=111
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&amp

Note the outputs are case-sensitive ASCII sorted, with trailing and duplicate/redundant ampersands cleaned.

Doing it in perl with internal reads and sorts is also a lot faster.

real    0m0.170s
user    0m0.046s
sys     0m0.092s

the old version

I see no more elegant way than a brute-force double pass, though you can at least eliminate redundant comparisons in the nested loop.

lst=( $( sort -ru x ) ) # unique reverse sort once to eliminate simple dups

for (( ndx1=0; ndx1<${#lst[@]}-1; ndx1++ ))       # walk thru once in outer loop
do [[ -n "${lst[ndx1]}" ]] || continue            # ignore removed
   for (( ndx2=ndx1+1; ndx2<${#lst[@]}; ndx2++ )) # inner skips prev, no redux
   do case "${lst[ndx1]}" in                      # case statement string match
      "${lst[ndx2]}"*) unset lst[ndx2] ;;         # remove shorter versions
                    *) continue 2      ;;         # no match, skip ahead
      esac
   done
done

printf "%s\n" "${lst[@]}"                         # print out what's left

I sort uniquely in reverse order to eliminate simple dups and set up comparisons, and store to an array for easy nested looping.

The outer loop walks the array once; it doesn't bother with the last record, because the inner loop will handle that. The inner loop starts at the record after the current one from the outer loop - no reason to check previous again since they are sorted.

Since the inner loop removed records, the outer loop skips checking entirely if the outer key record at the indicated index is empty.

The case statement checks each record after the current one from the outer loop. If the inner key is contained in the current outer loop key record, the shorter version is removed from the array with unset, and the loop proceeds to the next record to check that.

When the inner loop record is no longer a part of the outer loop key, we know we have moved past the relevant records (since they are sorted), so we skip checking the rest of the list pointlessly and move on to the next outer key record with continue 2.

This moving window of relevant records should do minimal wasted work.

Finally, looks like i've done it) for this test file:

$ cat file2
test?foo
test?bar
test?foo
test2?foo
test2?baz
test2?foo&bar
test2?baz
test?foo&bar
test?bar&foo
test2?bar&foo
test3?bar
test3?foo&bar&baz
test2?foo&bar&baz
test?foo&bar&baz

Script

#!/bin/bash
declare -A resorces
raw=( $(sort -u $1) )
for url in "${raw[@]}"; { resorces[${url//\?*}]+=" ${url//*\?}"; }
for res in "${!resorces[@]}"; {
    list=( ${resorces[$res]} )
    for i in "${!list[@]}"; {
        par=${list[$i]}
        unset list[$i]
        [[ ${list[@]} =~ $par ]] || result+=("$res?$par")
    }
}
printf '%s\n' "${result[@]}"

Result

$ ./test2 file2
test2?bar&foo
test2?foo&bar&baz
test3?foo&bar&baz
test?bar&foo
test?foo&bar&baz

And for this test file:

$ cat file
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&docID=111
http://grouplogic.com:80/news-events/index.cfm?fa=viewNews&ID=390
http://grouplogic.com:80/public/quickpoll/index.cfm?fuseaction=quickPollResults&QuestionID=8
http://grouplogic.com:80/store/index.cfm?cfid=11812682&cftoken=26157811&fa=conre
http://grouplogic.com:80/store/index.cfm?fa=PrtSlt&id=532&prTpID=5&
http://grouplogic.com:80/store/index.cfm?upTp=2&fa=upgrade&UpNewType=2&prTpID=5&&ptype=FS
http://grouplogic.com:80/news-events/index.cfm?fa=viewRelease&ID=21&prod=2
http://grouplogic.com:80/content/index.cfm?fuseaction=faq_list&ProdID=1&archive=1
http://grouplogic.com:80/content/index.cfm?ID=103
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&amp
http://grouplogic.com:80/knowledge/index.cfm?fuseaction=view&docID=10
http://grouplogic.com:80/content/index.cfm?ID=123
http://grouplogic.com:80/content/index.cfm?ID=123
http://grouplogic.com:80/content/index.cfm?ID=123
http://grouplogic.com:80/content/index.cfm?ID=123&foo=bar
http://grouplogic.com:80/content/index.cfm?ID=123&foo=bar

Result

$ ./test2 file
http://grouplogic.com:80/content/index.cfm?fuseaction=faq_list&ProdID=1&archive=1
http://grouplogic.com:80/content/index.cfm?ID=103
http://grouplogic.com:80/content/index.cfm?ID=123&foo=bar
http://grouplogic.com:80/public/quickpoll/index.cfm?fuseaction=quickPollResults&QuestionID=8
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&amp
http://grouplogic.com:80/Knowledge/index.cfm?fuseaction=view&docID=111
http://grouplogic.com:80/store/index.cfm?cfid=11812682&cftoken=26157811&fa=conre
http://grouplogic.com:80/store/index.cfm?fa=PrtSlt&id=532&prTpID=5&
http://grouplogic.com:80/store/index.cfm?upTp=2&fa=upgrade&UpNewType=2&prTpID=5&&ptype=FS
http://grouplogic.com:80/knowledge/index.cfm?fuseaction=view&docID=10
http://grouplogic.com:80/news-events/index.cfm?fa=viewNews&ID=390
http://grouplogic.com:80/news-events/index.cfm?fa=viewRelease&ID=21&prod=2

I see following algorithm to do this (unfortunately I don't know how to implement it):

First you sort your file, alphabetically.

Then you read your file, line by line, and if a line is a substring of the following line, it is not to be put in the resulting file.