Import Logs from S3 to Athena using Regex

Question

I have exported CloudWatch logs to S3 and now want to import those logs to Athena. The format of the logs is as follow (pasted only one log for reference):

2021-07-30T14:30:22.937Z    RequestId   INFO    {"_logLevel":"debug","msg":"Start: Calling All the Data Associates Function","timestamp":1627655422937,"EventSubCategory":"AppSyncService","API":"AppSyncService","function":"XXXXXXXXXXXXXXXXX","Correlation_Id":"XXXXXXXXXXXXXXXXX"}

I am using a regular expression to import the log and using the following query to create the table.

CREATE EXTERNAL TABLE IF NOT EXISTS test1 (
  `time` string
  `requestid` string
  `loglevel` string
  `message` string
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  'serialization.format' = '1',
  'input.regex' = '^(.*?)	(.*?)	(.*?)	([\s\S]*?)
'
)
LOCATION 's3://logs/test/'
TBLPROPERTIES ('has_encrypted_data'='false');

Regular Expression:

^(.*?)	(.*?)	(.*?)	([\s\S]*?)

There are four columns in the table and the regular expression is also creating four groups and working as per my expectation. However, we still get empty table as result.

Can anyone please help to resolve this issue?

user_1109 · Answer

I think your problem is that you need to double-escape things in the regex, and you also should not match on a newline at the end, but $. Try this pattern:

'input.regex' = '^(.*?)\t(.*?)\t(.*?)\t([\s\S]*?)$'

You can see an example in the official docs.

Also, the pattern [\s\S] could be replaced by . (\S means everything not matched by \s, so together they match anything).

An alternative to the regex serde is Grok, which is less error prone to write. Using the Grok serde I think this table would work for you:

CREATE EXTERNAL TABLE IF NOT EXISTS test1 (
  `time` string
  `requestid` string
  `loglevel` string
  `message` string
)
ROW FORMAT SERDE 'com.amazonaws.glue.serde.GrokSerDe'
WITH SERDEPROPERTIES (
  input.format' = '%{TIMESTAMP_ISO8601:time} %{NOTSPACE:requestid} %{NOTSPACE:loglevel} %{NOTSPACE:message}'
)
LOCATION 's3://logs/test/'

Grok patterns are much easier to read. Check out the documentation and the built-in patterns for more info.

Import Logs from S3 to Athena using Regex

1 Answers