Parse a log file and create a CVS report with tables in python. The code keeps returning errors for some reasons

Viewed 41

I'm stuck for more than a week already pleasee help
I don't understand why it doesn't work

The assignment:
Imagine your company uses a server that runs a service called ticky, an internal ticketing system. The service logs events to syslog, both when it runs successfully and when it encounters errors. The service's developers need your help getting some information from those logs so that they can better understand how their software is used and how to improve it. So, for this lab, you'll write some automation scripts that will process the system log and generate reports based on the information extracted from the log files.

What you'll do Use regex to parse a log file Append and modify values in a dictionary Write to a file in CSV format Move files to the appropriate directory for use with the CSV->HTML converter

Here is the examples from the log file:

Jan 31 16:35:46 ubuntu.local ticky: ERROR: Timeout while retrieving information (oren)
Jan 31 16:53:54 ubuntu.local ticky: INFO: Commented on ticket [#3813] (mcintosh)`
Jan 31 16:54:18 ubuntu.local ticky: ERROR: Connection to DB failed (bpacheco)
Jan 31 17:15:47 ubuntu.local ticky: ERROR: The ticket was modified while updating (mcintosh)
Jan 31 17:29:11 ubuntu.local ticky: ERROR: Connection to DB failed (oren)
Jan 31 17:51:52 ubuntu.local ticky: INFO: Closed ticket [#8604] (mcintosh)

My code (which keeps returning errors):

#!/usr/bin/env python3

import re
import sys
import operator
import csv

error = {}
per_user = {}

with open('syslog.log') as file:
  lines = file.readlines()
  for line in lines:
    result = re.search(r"ticky: ([A-Z]*):? ([\w' ]*)[\[[#0-9]*\]?]? \(([a-z\.?]*)\)$", line)
    category, message, username = result.group(1), result.group(2), result.group(3)

    if category == "ERROR" and message not in error.keys():
      error[message] = 1
    elif category == "ERROR" and message in error.keys():
      error[message] += 1

    if category == "INFO":
      if username not in per_user.keys():
        per_user[username] = {}
        per_user[username]["INFO"] = 1
        per_user[username]["ERROR"] = 0
      else:
        per_user[username]["INFO"] += 1
    elif category == "ERROR":
      if username not in per_user.keys():
        per_user[username] = {}
        per_user[username]["INFO"] = 0
        per_user[username]["ERROR"] = 1
      else:
        per_user[username]["ERROR"] += 1

# sort the error dictionary by the number of errors from most common to least common
sorted_errors = sorted(error.items(), key=operator.itemgetter(1), reverse=True)

# sort the user dictionary by username
sorted_users = sorted(per_user.items(), key=operator.itemgetter(0))

updated_errors = dict(sorted_errors)
updated_users = dict(sorted_users)

file.close()

# create error_message.csv 
# insert column names as ("Error", "Count") at the zero index position of the sorted error dictionary
keys = ["Error", "Count"]
with open("error_message.csv", "w", newline='') as error_csv:
  ew = csv.DictWriter(error_csv, fieldnames=keys)
  ew.writeheader()
  ew.writerows(updated_errors)
 
# create user_statistics.csv
# insert column names as ("Username", "INFO", "ERROR")
keys = ["Username", "INFO", "ERROR"]
with open("user_statistics.csv", "w", newline='') as user_csv:
  uw = csv.DictWriter(user_csv, fieldnames=keys)
  uw.writeheader()
  uw.writerows(updated_users)

I don't really understand the error messages so....I am really lost.

The errors:

Traceback (most recent call last):
  File "ticky_check.py", line 54, in <module>
    ew.writerows(updated_errors)
  File "/usr/lib/python3.8/csv.py", line 157, in writerows
    return self.writer.writerows(map(self._dict_to_list, rowdicts))
  File "/usr/lib/python3.8/csv.py", line 147, in _dict_to_list
    wrong_fields = rowdict.keys() - self.fieldnames
AttributeError: 'str' object has no attribute 'keys'
1 Answers

I'm not sure to understand the expected output, but maybe you can use pandas.read_fwf ?

import pandas as pd
from io import StringIO

t = """Jan 31 16:35:46 ubuntu.local ticky: ERROR Timeout while retrieving information (oren)
Jan 31 16:53:54 ubuntu.local ticky: INFO Commented on ticket [#3813] (mcintosh)`
Jan 31 16:54:18 ubuntu.local ticky: ERROR Connection to DB failed (bpacheco)
Jan 31 17:15:47 ubuntu.local ticky: ERROR The ticket was modified while updating (mcintosh)
Jan 31 17:29:11 ubuntu.local ticky: ERROR Connection to DB failed (oren)
Jan 31 17:51:52 ubuntu.local ticky: INFO Closed ticket [#8604] (mcintosh)"""

names=["SERVER", "INFO", "ERROR"]
df = pd.read_fwf(StringIO(t), header=None, names=names)
# or df = pd.read_fwf(r'path_to_your_logfile.txt', header=None, names=names)
df['USERNAME'] = df['ERROR'].str.extract('.*\((.*)\).*')

>>> display(df)

enter image description here

Related