Consolidated RegEx to parse syslog data

Question

Goal

I am trying to craft a RegEx that will parse out specific data from various syslog entries that contain subtle differences in logged content. While I am able to accomplish my goal using multiple RegEx statements, if possible, I would like to combine these statements into a single consolidated RegEx.

Log entries

The main issue I'm having is that some log entries have a URL that needs to be parsed to a named group and other log entries do not have any URL. Examples of these two different log entries are provided below.

Entry with URL

Nov  3 11:33:04 host1 postfix/smtpd[12812]: NOQUEUE: reject: RCPT from 178.red-83-59-180.dynamicip.rima-tde.net[83.59.180.178]: 554 5.7.1 Service unavailable; Client host [83.59.180.178] blocked using b.barracudacentral.org; http://www.barracudanetworks.com/reputation/?pr=1&ip=83.59.180.178; from= to= proto=ESMTP helo=<178.red-83-59-180.dynamicip.rima-tde.net>

Entry without URL

Nov  2 16:01:25 host1 postfix/smtpd[31667]: NOQUEUE: reject_warning: RCPT from mail1.sendersrv.com[185.3.229.125]: 554 5.7.1 Service unavailable; Client host [185.3.229.125] blocked using bl.spamcop.net; from= to= proto=ESMTP helo=

RegEx statements

In the RegEx statements that follow, the first two are what I currently use for each of the previous log messages. The third RegEx is my attempt at consolidating these both into a single RegEx that will parse data from either log message. My attempt was to use a conditional statement that would basically check for the existence of http(s) and if found, then to parse the URL to a named group. If http(s) was not found, then it would parse out everything until the next RegEx token.

The issue is that when I test the RegEx against a log entry that has a URL, the RegEx does not seem to find http(s) despite this token being set as optional (i.e. using the ? quantifier). However, if I remove the ? quantifier, it does find http(s) and then parses the URL as desired. However, without the quantifier, the RegEx does not work with log entries that do not have a URL.

Parse entries with URL

^(?P.+) host1 postfix.+RCPT from (?P.+)$$(?P[0-9\.]+)$$:.+blocked using (?P.+);.+https?:/{2}(?P.+);\s.+\sto=\<(?P.+)>.+$

Parse entries without URL

^(?P.+) host1 postfix.+RCPT from (?P.+)$$(?P[0-9\.]+)$$:.+blocked using (?P.+);\s.+\sto=\<(?P.+)>.+$

Attempt at consolidating RegEx

^(?P.+) host1 postfix.+RCPT from (?P.+)$$(?P[0-9\.]+)$$:.+blocked using (?P.+)(?<=[a-z]);.+(https?:/{2})?(?(5)(?P.+)|.+)to=\<(?P.+)>.+$

I'm sure the issue is my misunderstanding as to how the conditional statements and the ? quantifier works.

user_5424988 · Answer

Looking at your patterns, the email address for to: is between tags < and > but due to the formatting in the question they are not shown.

The parts in your pattern like .+ first match until the end of the string, and will then backtrack and try to match the rest of the pattern.

You can make the pattern a bit more performant making the parts that you want and know more specific.

For the datetime, you can make the pattern match the specified format instead of .+ using ^(?P[A-Z][a-z]{2}\s+\d{1,2}\s* \d{1,2}:\d{1,2}:\d{1,2})
For (?P[^;]+) and (?P[^;]+) you can use a negated character class matching any char except ;
For group (?P[^<>\s]+) you can exclude matching tags.
To make match the url, instead of using a condition you can make the group optional using ?

For example

^(?P[A-Z][a-z]{2}\s+\d{1,2}\s* \d{1,2}:\d{1,2}:\d{1,2}) host1 postfix\b.*? RCPT from (?P.*?)$$(?P[0-9\.]+)$$:.*? blocked using (?P[^;]+);(?:.+?https?://(?P[^;]+);)?\s.*? to=[^<]*<(?P[^<>\s]+)>

See a regex demo.

user_13525512 · Answer

Have you tried to test your regex on page like regex101?
to=\<(?P.+)> doesn't seem to match your examples. You should either remove <> or replace to with helo. Be careful to make your quantifier lazy after blkList otherwise you might catch too much text.
You can then make your url optional with ? and it should work in both cases:

^(?P.+) host1 postfix.+RCPT from (?P.+)$$(?P[0-9\.]+)$$:.+blocked using (?P.+?);(.+https?:/{2}(?P.+);\s)?.+\sto=(?P.+?)\s.*$

user_15250012 · Answer

One approach would be to replace in the first regex .+https?:/{2}(?P.+); with (?:.+https?:/{2}(?P.+);)? where ?: indicates that it is a non-capturing group and the ? at the end means that it is optional.

However, it still does not work because .+ is greedy, so use lazy .+? instead.

Final regex:

^(?P.+?) host1 postfix.+?RCPT from (?P.+?)$$(?P[0-9\.]+)$$:.+?blocked using (?P.+?);(?:.+?https?:/{2}(?P.+?);)?\s.+?\sto=\<(?P.+?)>.+?$

https://regex101.com/r/QkmXWz (to see it in action)