Create regex pattern for fluentd

John777 0
Viewed 9

I'm using Fluentd configuration and regexp parser to parse the logs.

This are my original logs in fluentd:

2022-09-22 18:15:09,633 [springHikariCP housekeeper    ] DEBUG HikariPool                     - springHikariCP - Fill pool skipped, pool is at sufficient level.
2022-09-22 18:15:14,968 [ringHikariCP connection closer] DEBUG PoolBase                       - springHikariCP - Closing connection com.mysql.cj.jdbc.ConnectionImpl@7f535ea4: (connection has passed maxLifetime)

I want to create json format for the above logs with regexp like this:

{
    "logtime": "2022-09-22 18:15:09,633",
    "Logger Name": "[springHikariCP housekeeper    ]",
    "Log level": "DEBUG",
    "message": "HikariPool                     - springHikariCP - Fill pool skipped, pool is at sufficient level"
}

{
    "logtime": "2022-09-22 18:15:09,633",
    "Logger Name": "[ringHikariCP connection closer]",
    "Log level": "DEBUG",
    "message": "PoolBase                       - springHikariCP - Closing connectioncom.mysql.cj.jdbc.ConnectionImpl@7f535ea4: (connection has passed maxLifetime)"
}

Can someone with Ruby expirience help me to create Ruby regex for this logs above

fluentd ruby
1 Answers

The regex should be straightforward, e.g.

(?'logtime'\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s(?'logname'\[[^\]]*\])\s(?'loglevel'\w+)\s(?'message'.*)

Online Test

And then, reassemble the parts using a substitution (gsub, see example below) or putting a new string together from the parts (match) :

re = /(?'logtime'\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s(?'logname'\[[^\]]*\])\s(?'loglevel'\w+)\s(?'message'.*?$)/m
str = '2022-09-22 18:15:09,633 [springHikariCP housekeeper    ] DEBUG HikariPool                     - springHikariCP - Fill pool skipped, pool is at sufficient level.
2022-09-22 18:15:14,968 [ringHikariCP connection closer] DEBUG PoolBase                       - springHikariCP - Closing connection com.mysql.cj.jdbc.ConnectionImpl@7f535ea4: (connection has passed maxLifetime)'

str.gsub(re) do |match|
  puts "{
    \"logtime\": \"#{$~[:logtime]}\",
    \"Logger Name\": \"#{$~[:logname]}\",
    \"Log level\": \"#{$~[:loglevel]}\",
    \"message\": \"#{$~[:message]}\"
}"
end
wp78de0
Related