I'm trying to format scrapy's logs in JSON format. For that I have tried multiple things,
- First, to use
"LOG_FORMAT": "%(asctime)s [%(name)s]": "%(levelname)s: %(message)s",
But with that, I still have to add brackets at the beginning and the end of the file + handle multi-line logs (remove all \n). And even with that, it didn't work because I couldn't manage to change the format of the Tracebacks
The second idea was to rewrite the logs in the wanted format in the
def closed()doing something like that:"LOG_FORMAT": "%(asctime)s [%(name)s]:: %(levelname)s: %(message)s", ... def closed(self, reason): import json dict_json = {} json_logs = open(self.custom_settings["LOG_FILE"].replace(".log", ".json"), "w") with open(self.custom_settings["LOG_FILE"]) as file: lines = file.readlines() for line in lines: if f"{datetime.datetime.today().strftime('%Y-%m-%d')} " in line: key, value = line.strip().split("::", 1) dict_json[key] = value else: dict_json[key] += "\n" + line.strip() dict_json["stats"] = str(self.crawler.stats.get_stats()) json.dump(dict_json, json_logs, indent=4, sort_keys=False) json_logs.close()
It does work as expected but it doesn't make sense to write logs than rewrite them with the expected format... Does anyone think of a smarter way to do this ? Thanks in advance