here is how looks my datafiles (from directory) all is a huge (mess) log file pit containing different datas
so I have to get from that some data
here how it looks if I concatenate a few data
2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_DDD;RRRR_SSDDDD
2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_III;FT_RECEPTION
2022-08-31 23:51:05,560; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<7d4e2260-bf7e-4d81-9cd3-be7e4decaaae>;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; END TRANSFER FROM SOURCE
2022-08-31 23:51:07,233; example_data;;;example_data;TYPE_DDD;CON_CheckMeshArea_Output
2022-08-31 23:51:07,233; example_data;;;example_data;TYPE_III;APP_REQload rate = 171.142578125 KB/s ;
2022-08-31 23:51:07,521; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;;<Source: PPPPP-CON;CON_CheckMeshArea_Output>; ;<Destination:PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea_Output>;;<7740db78-8
05ad9427ab7>; BEGIN UPLOAD TO DESTINATION
2022-08-31 23:51:07,523; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;example_data;TYPE_DDD;CON_CheckMeshArea_Output
2022-08-31 23:50:51,990; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;<cd0fea11-de93-482c-b745-4336688e13ed>;;<Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea>; BEGIN TRANSFER FROM SOURCE
2022-08-31 23:50:52,118; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;<Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea>; ;<Destination:PPPPP-CON;CON_CheckMeshArea>;;<f13b8416-1a13-4212-aac8-
BEGIN UPLOAD TO DESTINATION
2022-08-31 23:50:52,359; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;Uploaded size = 1 KB ; Upload duration = 0.008 seconds ; Upload rate = 245.4833984375 KB/s ;
2022-08-31 23:50:52,359; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;<Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea>; ;<Destination:PPPPP-CON;CON_CheckMeshArea>;;<f13b8416-1a13-4212-aac8-
END UPLOAD TO DESTINATION
2022-08-31 23:50:52,377; example_data;cd0fea11-de93-482c-b745-4336688e13ed;;;<cd0fea11-de93-482c-b745-4336688e13ed>;;<Source: PPPPP-GGGMMMM-VVV-VVVVVV;CON_CheckMeshArea>; END TRANSFER FROM SOURCE
2022-08-31 23:51:01,920; example_data;;;example_data;TYPE_DDD;RRRR_SSDDDD
2022-08-31 23:51:01,920; example_data;;;example_data;TYPE_III;APP_REQ
2022-08-31 23:51:01,965; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<7d4e2260-bf7e-4d81-9cd3-be7e4decaaae>;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; BEGIN TRANSFER FROM SOURCE
2022-08-31 23:51:01,966;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP input data;server=p-APPSRV-82.SOMEDOMAIN port=990 user=service-p-PPPPPlication-0015 binaryTransfer=true passiveMode=true bufferSiz
cit=true protocol=TLS
2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply code=226
2022-08-31 23:51:03,575; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;Downloaded size = 135895 KB ; Download duration = 1.045 seconds ; Download rate = 130043.72009569379 KB/s ;
2022-08-31 23:51:03,647; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; ;<Destination:PPPPP-GGGIP-MMMM;RRRR_SSDDDD>;;<7b19f9ba-203d-4479-96b2-cc6d664bffcd>;
2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply string=226 Directory send OK.
2022-08-31 23:51:07,279; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;;<8841c2be-ee3a-4508-9c50-f817e5c7dfc8>;;<Source: PPPPP-CON;CON_CheckMeshArea_Output>; BEGIN TRANSFER FROM SOURCE
2022-08-31 23:51:07,479; example_data;8841c2be-ee3a-4508-9c50-f817e5c7dfc8;;Downloaded size = 0 KB ; Download duration = 0.004 seconds ; Down
from here I wanna extract for each Source/Destination couple the upload/download rate/size/duration & id's (the hexadecimal parts) So I had in my idea to
- sort by ID
- sort each by date
- make a specific output file for each
- then read each file for correlation to date/source/destination/size/rate/duration
Si my idea was to run a loop to extract that from a regexp & made it readable as a csv but I had memory errors
(the host has only 4Go)
I tryied many things like
like
import os
import re
import string
import sys
import datetime
datalogpath = sys.argv[1] # get the path of log files to extract datas
startdatestamp = sys.argv[2] # get a chronologic start marker
enddatestamp = sys.argv[3] # get a chronologic start marker
outputfilename = sys.argv[4] # output file will be csv formated results
outputfilename2 = 'testsort.txt' # output file will be csv formated results
searchpattern = re.compile('linked to the consume|size|rate|type_|source|dest') # regexp to filter from logs directory
err_occur = [] # just to manage error on empty line while reading the logs files
t_fmt = '%Y-%M-%D %H:%m:%S,%f' # timestamp format
line_array = [] # line buffer
for filename in os.listdir(datalogpath):
with open(os.path.join(datalogpath, filename)) as in_file:
for line in in_file:
#found = searchpattern.match(line)
found = searchpattern.search(line)
if found :
# STEP2
line = line.replace('|', ' - ')
line = line.replace(' - ', ';')
line = line.replace(' : ', ';')
line = line.replace(' INFO ', ';')
line = line.replace(' DEBUG ', ';')
line = line.replace('>', '>;')
line = line.replace('<', ';<')
line_array += line # keep line
# end open
sorted_line_array = sorted(line_array,key=lambda line: datetime.strptime(line_array.split(' ')[2], t_fmt))
It might be run by : python script.py ../data/ 20220810 20220813 out.txt
I guess a better method would be to make it read/grep all of a date as a dictionary -> then sort it -> then write specific file for it ? if so how can I do that ? (I m more or less new to python ; an short Example would be appreciated)
Currently I got "memory error" whatever I change my algo/method
The ideal result I would get is separated files for each ID within all sorted by date lines matching the ID (as very final result)
so expectation is to get separated files like :
7d4e2260-bf7e-4d81-9cd3-be7e4decaaae
cd0fea11-de93-482c-b745-4336688e13ed
8841c2be-ee3a-4508-9c50-f817e5c7dfc8
etc...
each containing all matching lines about that ID sorted by date/time
so for example for the 7d4e2260-bf7e-4d81-9cd3-be7e4decaaae file it will be
2022-08-31 23:51:01,965; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<7d4e2260-bf7e-4d81-9cd3-be7e4decaaae>;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; BEGIN TRANSFER FROM SOURCE
2022-08-31 23:51:01,966;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP input data;server=p-APPSRV-82.SOMEDOMAIN port=990 user=service-p-PPPPPlication-0015 binaryTransfer=true passiveMode=true bufferSiz
2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply code=226
2022-08-31 23:51:02,341;example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;CPAPP;uploadFile;setBufferSize;reply string=226 Directory send OK.
2022-08-31 23:51:03,575; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;Downloaded size = 135895 KB ; Download duration = 1.045 seconds ; Download rate = 130043.72009569379 KB/s ;
2022-08-31 23:51:03,647; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; ;<Destination:PPPPP-GGGIP-MMMM;RRRR_SSDDDD>;;<7b19f9ba-203d-4479-96b2-cc6d664bffcd>;
2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_DDD;RRRR_SSDDDD
2022-08-31 23:51:05,491; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;example_data;TYPE_III;FT_RECEPTION
2022-08-31 23:51:05,560; example_data;7d4e2260-bf7e-4d81-9cd3-be7e4decaaae;;;<7d4e2260-bf7e-4d81-9cd3-be7e4decaaae>;;<Source: FFFFFFFF-GGGGGG-PPPPPlicationS;RRRR_SSDDDD>; END TRANSFER FROM SOURCE
currently here is what I got is looking like :
[workdir]$ python loganalysze.fpu.py 2022-08-10 2022-08-13 ../data_Z2/
Traceback (most recent call last):
File "loganalysze.fpu.py", line 34, in <module>
found = searchpattern.search(line)
MemoryError
[workdir]$