Extract data from unix log file, construct JSON and perform post request using curl

user_1987588 0
Viewed 112

My overall task is constantly to collect data from UNIX system log file, filter it, prepare a json payload based on the filtered data and process the data by sending a post api call to another server.

I wonder if that can be done using let's say shell script to monitor the log file with tail, filter with grep to get the specific lines dumpted in another file. With cronjob to run another script which contruct a .json and send curl request with the json to external server.

Some details:

In the log file - connector.log I am interested in lines like:

2020-09-16T15:14:37,337 INFO  (tomcat-http--131) [tenant-test;-;138.188.247.4;] com.vmware.horizon.adapters.passwordAdapter.PasswordIdpAdapter - Login: user123 - SUCCESS

These lines, I can collect by the below command:

tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS'

and probably dump them into a file:

tailf connector.log | grep 'PasswordIdpAdapter - Login\|FAILURE\|SUCCESS' > log_data.txt

I wonder at this point, is it possible to extract only specific fields from a line(not the whole line) from the connector.log , so one line in log_data.txt to look like(1, 4, 6, 7, 8):

1 2020-09-29T07:15:13,881 [tenant1;usrname@tenant1;10.93.231.5;] - username - SUCCESS

From that point, I need to write a script(maybe could be run by cronjob every minute)/or a command to construct the below json and send the request. One line - one request.

This is the example of the json:

{
   "timestamp": "2020-09-16T15:24:35,377",
   "tenant_name": "tenant-test",
   "log_type": "SERVICE",
   "log_entry": "Login: user123 - SUCCESS"
}

The field values that should be replaced already exist in the log line: timestamp(the 1st field, e.g. 2020-09-16T15:14:37,337), tenant_name(the 1st part of the 4th field, tenant-test) and the log_entry(the last four fields, e.g. Login: user123 - SUCCESS).

When the json is constructed, I'll send it by:

curl --header "Content-Type: application/json" --request POST --data \
  $payload http://myservert:8080/api/requests

What is not clear to me, this script to get the data line by line from log_data.txt e.g.

and populate some of the fields to create the .json and send it to the server.

Thanks for your answers in advance, Petko

curl shell unix linux json
1 Answers

Thanks @shellter for the awk idea. So, bash, awk, grep, cat, cut and curl did the job. I've created a cronjob to execute the bash script on 5 min interval.

The script gets the last 5mins of log data, dump it to another file, reads the filtered data, prepare the payload and then executes the API call. Maybe it is stupid but it works.

#!/bin/bash

MONITORED_LOG="/var/logs/test.log"
FILTERED_DATA="/tmp/login/login_data.txt"
REST_HOST="https://rest-host/topics/logs-"

# dump the last 5 mins of log data(date format: 2020-09-28T10:52:28,334)
# to a file, filter for keywords FAILURE\|SUCCESS and NOT having 'lookup|SA' 
# an example of data record taken: 1 2020-09-29T07:15:13,881 [tenant1;usrname@tenant1;10.93.231.5;] - username - SUCCESS

awk -v d1="$(date --date="-5 min" "+%Y-%m-%dT%H:%M:%S")" -v d2="$(date "+%Y-%m-%dT%H:%M:%S")" '$0 > d1 && $0 < d2' $MONITORED_LOG | grep 'FAILURE\|SUCCESS' | grep -v 'lookup\|SA-' | awk '{ print $2, $3, $5, $7}' | uniq -c > $FILTERED_DATA

## loop through all the filtered records and send an API call
cat $FILTERED_DATA | while read LINE; do

## preparing the variables
timestamp=$(echo $LINE | cut -f2 -d' ')
username=$(echo $LINE | cut -f5 -d' ')
log_entry=$(echo $LINE | cut -f7 -d' ')
# get the tenant name, split by ; and remove the first char [ 
tenant_name=$(echo $tenant_name | cut -f1 -d';')
tenant_name="${tenant_name:1}"
    
# preparing the payload
payload=$'{"records":[{"value":{"timestamp":"'
payload+=$timestamp
payload+=$'","tenant_name":"'
payload+=$tenant_name
payload+=$'","log_entry":"'
payload+=$log_entry
payload+=$'"}}]}'
echo 'payload: ' $payload

# send the api call to the server with dynamic construction of tenant name
curl -i -k  -u 'api_user:3494ssdfs3' --request POST --header "Content-type:application/json" --data "$payload" "$REST_HOST$tenant_name"

done

user_19875880
Related