How to check and get lines of a log file that has changed since last run?

noob_coder 0
Viewed 19

I have a task where I am reading a large file(~ 2 GB) and the file is getting appended every few seconds and have a cron job set that runs every 5 mins. Now this cron job checks and gets only the "appended" lines and looks for a pattern in those appended lines and stores the lines containing that pattern in some kind of file/data structure.

now my idea around this is

  1. check for the file and see if it has changed since last run, if "yes" get what lines have been appended to it.
  2. Look for the pattern in the appended lines and get the lines and store them somewhere. I am clear on the code in step2, but I'm stuck in step1.

Since the file is appending, how do I look for lines that are appended. Can I use something like "watch -n 5 mylog.txt" and run this via subprocess module and see what lines have changed or is there any other good way to do it. OR

see what was the line number of the last line from last Run and check the line number in the next run and print lines between these 2, but how do I put this to code. I have no clue.

Please someone help.

python
1 Answers

Keep a separate file, line_count.txt, that has one value: the previous number of lines seen in the large file. When you run the cron job your script should open this file to see what was the previous line count, and then start reading from that point until the end of the file. Your script should also keep track of how many new lines it reads so that after your script is done it will update line_count.txt by adding up the previous line count with the number of new lines you have read.

Psuedo code:

# Begin script
open line_count.txt as line_count_file
prev_line_count = int(line_count_file.read())

analyze from prev_line_count + 1 to end of file

new_line_count = prev_line_count + num_new_lines
line_count_file.write(str(new_line_count))
JRose0
Related