Parse nginx access log and extract IP, detect geo location for each parsed IP

user_4456923 0
Viewed 958

I have question about parsing nginx access log,

have error in this code <<< grep "pagename" <<< "$line0"

not work grep "pagename" 

while IFS= read -r line0
do    
ipList=$( grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b'
<<< grep "pagename" <<< "$line0")
for oneIP in $ipList
do
    curl -s  "http://ipinfo.io/$oneIP"
done

done < /var/log/nginx/access.log

Thanks.

geolocation nginx curl bash
1 Answers

The syntax <<< grep "pagename" <<< "$line0" is slighly off, <<<X inputs the string X to the stdin.

The syntax you want is the process substitution syntax (combined with a redirection-arrow):

< <(grep "pagename" <<< "$line0")

Here's also a simpler solution:

#! /bin/bash
while read -r oneIP; do
    curl -s "http://ipinfo.io/$oneIP"
done < <(grep "pagename" /var/log/nginx/access.log \
         | grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b')

The result of the first grep is piped to the second, and that stream of ips is then redirected to the while-loop which runs the curl for each of the ips.

You could also parallelize this with something like gnu parallel:

#! /bin/bash
grep "pagename" /var/log/nginx/access.log \
| grep -oP '\b(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))\b' \
| parallel echo curl -s "http://ipinfo.io/"{}
user_24163760
Related