How to extract text from access log?

user_12224391 0
Viewed 203

I am very new in this. I am trying to extract some text from my access log in a new file.
My log file is like this:

111.111.111.111 - - [02/Jul/2021:18:35:19 +0000] "GET /api/items HTTP/2.0" 304 0 "https://example.com/some/text/call-log?roomNo=5003" "Mozilla etc etc etc etc"
111.111.111.111 - - [02/Jul/2021:20:35:19 +0000] "GET /api/items HTTP/2.0" 304 0 "https://example.com/some/text/resevation-log?roomNo=4003" "Mozilla etc etc etc etc"

I want to extract in below format in a new file.

02/Jul/2021:18:35:19 +0000, call-log, 5003
02/Jul/2021:20:35:19 +0000, resevation-log, 4003

Till now I have managed to do this basic awk command:

awk '{print $4,$5,",",$11}' < /file.log

Which gives me the below output:

[02/Jul/2021:18:35:19 +0000] , "https://example.com/some/text/call-log?roomNo=5003"
unix-text-processing awk
3 Answers
$ cat tst.awk
BEGIN {
    FS="[[:space:]]*[][\"][[:space:]]*"
    OFS = ", "
}
{
    n = split($6,f,"[/?=]")
    print $2, f[n-2], f[n]
}


$ awk -f tst.awk file
02/Jul/2021:18:35:19 +0000, call-log, 5003
02/Jul/2021:20:35:19 +0000, resevation-log, 4003

The above uses the following way to split the input in your question into fields using any POSIX awk:

$ cat tst.awk
BEGIN {
    FS="[[:space:]]*[][\"][[:space:]]*"
    OFS = ","
}
{
    print
    for (i=1; i<=NF; i++) {
        print "\t" i, "<" $i ">"
    }
    print "-----"
}


$ awk -f tst.awk file
111.111.111.111 - - [02/Jul/2021:18:35:19 +0000] "GET /api/items HTTP/2.0" 304 0 "https://example.com/some/text/call-log?roomNo=5003" "Mozilla etc etc etc etc"
        1,<111.111.111.111 - ->
        2,<02/Jul/2021:18:35:19 +0000>
        3,<>
        4,<GET /api/items HTTP/2.0>
        5,<304 0>
        6,<https://example.com/some/text/call-log?roomNo=5003>
        7,<>
        8,<Mozilla etc etc etc etc>
        9,<>
-----
111.111.111.111 - - [02/Jul/2021:20:35:19 +0000] "GET /api/items HTTP/2.0" 304 0 "https://example.com/some/text/resevation-log?roomNo=4003" "Mozilla etc etc etc etc"
        1,<111.111.111.111 - ->
        2,<02/Jul/2021:20:35:19 +0000>
        3,<>
        4,<GET /api/items HTTP/2.0>
        5,<304 0>
        6,<https://example.com/some/text/resevation-log?roomNo=4003>
        7,<>
        8,<Mozilla etc etc etc etc>
        9,<>
-----

That would fail if any of your quoted fields can contain [, ], or an escaped ", none of which exist in your example but if they can happen then include them in the example in your question.

user_17450010

This awk can extract the text:

awk -v FS='[][/?="]' -v OFS=',' '{print $2"/"$3"/"$4,$16,$18}' file
02/Jul/2021:18:35:19 +0000,call-log,5003
02/Jul/2021:20:35:19 +0000,resevation-log,4003
user_142594650

Another way of doing this using AWK is:

awk '{split($11, A, /\/+|"|(\?roomNo=)/); print substr($4, 2), substr($5, 1, 5) ",", A[6] ",", A[7]}' file.log >> newFile.log

First part is splitting the URL field into an array using regex,
then printing the specific fields and array values
Lastly storing the logs into another file named newFile.log

Edit:
And yet another shortest and fastest one-liner based on the log output above is using sed: (preferred)

sed -E 's/\].+\/|\?roomNo=/, /g; s/^.+\[|".+$//g' file.log >> newFile.log

where the first substitution replaces ] "GET /api/items HTTP/2.0" 304 0 "https://example.com/some/text/ and ?roomNo= with a , and the second substitution removes the first and last part which are 111.111.111.111 - - [ and " "Mozilla etc etc etc etc"

user_162360440
Related