filter unique parameters from file

user_11157714 0
Viewed 133

i have file contains urls plus params like following

https://example.com/endpoint/?param1=123&param2=1212
https://example.com/endpoint/?param3=123&param1=98989
https://example.com/endpoint/endpoint3/?param2=123
https://example.com/endpoint/endpoint2/?param1=123
https://example.com/endpoint/endpoint2/
https://example.com/endpoint/endpoint5/"//i.example.com/00/s/Nzk5WDEwMjQ=/z/47IAAOSwBu5hXIKF

and i need to filter only urls with unique params the desired output

http://example.com/endpoint/?param1=123&param2=1212
https://example.com/endpoint/?param3=123&param1=98989
https://example.com/endpoint/endpoint3/?param2=123

i managed to filter only urls with params with grep grep -E '(\?[a-zA-Z0-9]{1,9}\=)'

but i need to filter params in the same time so i tried with awk with the same regex but it gives error

awk '{sub(\?[a-zA-Z0-9]{1,9}\=)} !seen[$0]++'

update

i am sorry for editing the desired output but when i tried the scripts i figured out that their a lot of carbege in my file need to filter too. i tried @James Brown with some editing and it looks good till the end line it dose not filter it unfortunately

awk -F '?|&' '$2&&!a[$2]++'

and to be more clear why the that output is good for me it chosed the 1 st line because it has at least param1 2nd line because it has at least param3 3 line because it has at least param2 the comparison method here is choose just unique parameter whatever it concatenate with others with & char or not

grep awk
3 Answers

EDIT: Following solution may help when query string has ? as well as & present in it and we want to consider both of them for removing duplicates.

awk '
/\?/{
  match($0,/\?[^&]*/)
  val=substr($0,RSTART,RLENGTH)
  match($0,/&.*/)
  if(!seen[val]++ && !seen[substr($0,RSTART,RLENGTH)]++){
    print
  }
}' Input_file

2nd solution: (Following solution may help when we don't have & parameters in query string) With your shown samples, please try following awk program.

awk 'match($0,/\?.*$/) && !seen[substr($0,RSTART,RLENGTH)]++' Input_file

OR above could be shorten to as follows:(as per Ed sir's suggestions):

awk 's=index($0,"?") && !seen[substr($0,s)]++' Input_file

Explanation: Simple explanation would be, using match function of awk which matches everything from ? to till end of line value. Then adding an AND condition to it to make sure we get only unique values out of all matched values in all lines.

user_58665800

Edited version after the reqs changes some:

$ awk -F? '{                   # ? as field delimiter
    split($2,b,/&/)            # split at & to get whats between ? and &
    if(b[1]!=""&&!a[b[1]]++)   # no ? means no $2
        print
}' file

Output as expected. Original answer was:

A short one:

$ awk -F? '$2&&!a[$2]++' file

Explained: Split records at ? (-F?) and if there is a second field ($2) and (&&) it is unique this far by counting the instances of the parameters in the array a (!a[$2]++), output it.

user_41623560

With gnu awk, you could also match the url till the first occurrence of the question mark, and then capture what follows using your initial pattern for the first parameter ([a-zA-Z0-9]{1,9}=[^&]+) followed by matching any character except the &

Then you can use the !seen[$0]++ part with the value of capture group 1.

awk '
match($0, /https?:\/\/[^?]+\?([a-zA-Z0-9]{1,9}=[^&]+)/, arr) && !seen[arr[1]]++
' file

Output

https://example.com/endpoint/?param1=123&param2=1212
https://example.com/endpoint/?param3=123&param1=98989
https://example.com/endpoint/endpoint3/?param2=123

Using awk you can check that the string starts with the protocol and contains a question mark.

Then to get the first parameter only, you can split on ? and & and use the second part of the split for seen

awk '
/^https?:\/\/[^?]*\?/ && split($0, arr, /[?&]/) > 1 && !seen[arr[2]]++
' file
user_54249880
Related