Read log file in R so that each row starts with timestamp

user_13174189 0
Viewed 135

I have a log file with such data inside:

2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 10 - Trying to upload data
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 66 - Trying to upload data
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidationXmlParser: 127 - No META-Only annotation
2020-07-28 14:48:00 (pool-2-thread-1id) DEBUG MessageWriter: 55 - Send message ErrorOutputMessage(super=NotificationOutputMessage(super=OutputMessage(type=null, messageId=116345, reqId=af24112))), error=ErrorOutputMessage.Error(code=400, text={
  "errors": [
    "Message type error"
  ]
})) to exchange FOS 
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidatorFactoryImpl: 578 - Scoped message interpolator.

I try to read that file in this way:

data <- readr::read_lines(file = "log_data.log", progress = FALSE)
log_df <- setDT(tibble::enframe(data, name = NULL))

But this dataframe looks like this:

              value
1   2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 10 - Trying to upload data
 
2   2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 66 - Trying to upload data
3   2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidationXmlParser: 127 - No META-Only annotation
4   2020-07-28 14:48:00 (pool-2-thread-1id) DEBUG MessageWriter: 55 - Send message ErrorOutputMessage(super=NotificationOutputMessage(super=OutputMessage(type=null, messageId=116345, reqId=af24112))), error=ErrorOutputMessage.Error(code=400, text={
5     "errors": [
6         "Message type error"
7     ]
8   })) to exchange FOS 
9   2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidatorFactoryImpl: 578 - Scoped message interpolator.

So as you see row number 4 splited into several rows, thought its one. How could i read this log file, so it understands that each row must start with timestamp? Should i use regular expressions somehow?

dataframe r regex
3 Answers

You can use regex to remove all new-lines not followed by a date:

log_string <- '2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 10 - Trying to upload data
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 66 - Trying to upload data
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidationXmlParser: 127 - No META-Only annotation
2020-07-28 14:48:00 (pool-2-thread-1id) DEBUG MessageWriter: 55 - Send message ErrorOutputMessage(super=NotificationOutputMessage(super=OutputMessage(type=null, messageId=116345, reqId=af24112))), error=ErrorOutputMessage.Error(code=400, text={
  "errors": [
    "Message type error"
  ]
})) to exchange FOS 
2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidatorFactoryImpl: 578 - Scoped message interpolator.'

readr::read_lines(
    stringr::str_replace_all(
        log_string,
    
        # Regular expression
        '(\\r?\\n|\\r)(?!\\d{4}-(\\d{2}[-: ]){5})', '')
)


[1] "2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 10 - Trying to upload data"                                                                                                                                                                                                                      
[2] "2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ResourceLoaderHelper: 66 - Trying to upload data"                                                                                                                                                                                                                      
[3] "2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidationXmlParser: 127 - No META-Only annotation"                                                                                                                                                                                                                    
[4] "2020-07-28 14:48:00 (pool-2-thread-1id) DEBUG MessageWriter: 55 - Send message ErrorOutputMessage(super=NotificationOutputMessage(super=OutputMessage(type=null, messageId=116345, reqId=af24112))), error=ErrorOutputMessage.Error(code=400, text={  \"errors\": [    \"Message type error\"  ]})) to exchange FOS "
[5] "2020-07-28 10:07:01 (pool-3-thread-5id) DEBUG ValidatorFactoryImpl: 578 - Scoped message interpolator."

(\\r?\\n|\\r) matches different versions of new-line.

(?!\\d{4}-(\\d{2}[-: ]){5}) is a negative lookahead that matches a datetime (copied from EvilSmurf's answer).

user_14986560

A two pass approach is possible:

  1. In Notepad++, use regex replace with the \R++(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ) pattern and some placeholder replacement, say, <L1nBre@k>. Start the regex engine to see how this expression works.
  2. Once the file is read in, revert linebreaks:
data <- gsub('<L1nBre@k>', '\n', data, fixed=TRUE)

\R++(?!\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2} ) explanation:

--------------------------------------------------------------------------------
  \R++                     line break sequence (1 or more times, as many
                           as possible), matched possessively
--------------------------------------------------------------------------------
  (?!                      look ahead to see if there is not:
--------------------------------------------------------------------------------
    \d{4}                    digits (0-9) (4 times)
--------------------------------------------------------------------------------
    -                        '-'
--------------------------------------------------------------------------------
    \d{2}                    digits (0-9) (2 times)
--------------------------------------------------------------------------------
    -                        '-'
--------------------------------------------------------------------------------
    \d{2}                    digits (0-9) (2 times)
--------------------------------------------------------------------------------
                             ' '
--------------------------------------------------------------------------------
    \d{2}                    digits (0-9) (2 times)
--------------------------------------------------------------------------------
    :                        ':'
--------------------------------------------------------------------------------
    \d{2}                    digits (0-9) (2 times)
--------------------------------------------------------------------------------
    :                        ':'
--------------------------------------------------------------------------------
    \d{2}                    digits (0-9) (2 times)
--------------------------------------------------------------------------------
                             ' '
--------------------------------------------------------------------------------
  )                        end of look-ahead
user_113298900

yes, i think regex would be a good way to go

which programming language are you using?

in python format the expression would be something like this:

\d{4}-(\d{2}[-: ]){5}(?P<your_data>[\s\S]*?)(?=\s*(\d{4}-(\d{2}[-: ]){5}|$))

I have made you an example here

UPDATE:

if you actually only want to match the lines with timestamps you can simplify the regex-pattern to this:

\d{4}-(?:\d{2}[-: ]){5}[^\n]*
user_39072580
Related