How to have scrapy 2.x logs in JSON format?

Viewed 67

I'm trying to format scrapy's logs in JSON format. For that I have tried multiple things,

  • First, to use "LOG_FORMAT": "%(asctime)s [%(name)s]": "%(levelname)s: %(message)s",

But with that, I still have to add brackets at the beginning and the end of the file + handle multi-line logs (remove all \n). And even with that, it didn't work because I couldn't manage to change the format of the Tracebacks

  • The second idea was to rewrite the logs in the wanted format in the def closed() doing something like that:

     "LOG_FORMAT": "%(asctime)s [%(name)s]:: %(levelname)s: %(message)s",
    
     ...
    
     def closed(self, reason):
         import json
         dict_json = {}
         json_logs = open(self.custom_settings["LOG_FILE"].replace(".log", ".json"), "w")
         with open(self.custom_settings["LOG_FILE"]) as file:
             lines = file.readlines()
             for line in lines:
                 if f"{datetime.datetime.today().strftime('%Y-%m-%d')} " in line:
                     key, value = line.strip().split("::", 1)
                     dict_json[key] = value
                 else:
                     dict_json[key] += "\n" + line.strip()
    
         dict_json["stats"] = str(self.crawler.stats.get_stats())
         json.dump(dict_json, json_logs, indent=4, sort_keys=False)
         json_logs.close()
    

It does work as expected but it doesn't make sense to write logs than rewrite them with the expected format... Does anyone think of a smarter way to do this ? Thanks in advance

0 Answers
Related