Conditionally merge adjacent list elements in Python

user_10394098 0
Viewed 307

I'm trying to parse a WhatsApp chat log and organize it into a DataFrame. I've managed to separate it into Date, Time, Name, and the Message. However, there are some newline messages that are continuations of the previous message and are in the "Date" column when I create the DataFrame. I want them to be appended to the previous message cell.

This is what the raw .txt file looks like (I made up some text to hide messages):

11/28/17, 10:00 AM - Bob: Lorem ipsum dolor sit amet
11/28/17, 10:00 AM - Marley: Yes!
11/28/17, 10:00 AM - Marley: consectetur adipiscing elit
11/28/17, 10:00 AM - Bob: Barely dude. BARELY
11/28/17, 10:01 AM - Bob: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
11/28/17, 10:14 AM - Marley: Ut enim ad minim veniam
11/28/17, 10:20 AM - Marley: quis nostrud exercitation
Duis aute irure dolor in
11/28/17, 10:31 AM - Bob: Hahaha proud
11/28/17, 10:31 AM - Bob: Can't imagine

As you can see, the message at 11/28/17, 10:20 AM is two lines. I want to append that extra message line to be with the message in the previous line. When I convert to a DataFrame, everything would belong in the correct column. Here's my code so far:

import pandas as pd

with open('whatsapp.txt', encoding="utf8") as f:
    mylist = list(f)
    df = pd.DataFrame(mylist)
    df = df[0].str.split(r'[,-]', 2, expand=True)
    df = df.rename(columns={0:"Date",1:"Time",2:"Name"})
    df = df.replace('\n','', regex=True)
    df[['Name','Message']] = df['Name'].str.split(':',1,expand=True)

My logic is before creating the DataFrame to find the list elements that do NOT start with '\d{1,2}/', then append those to the end of the previous element. Any ideas on how I can do this?

python-3.x python
4 Answers

Try this, working with me. 

from io import StringIO
import itertools
import re
import pandas as pd

a ="""11/28/17, 10:00 AM - Bob: Lorem ipsum dolor sit amet
11/28/17, 10:00 AM - Marley: Yes!
11/28/17, 10:00 AM - Marley: consectetur adipiscing elit
11/28/17, 10:00 AM - Bob: Barely dude. BARELY
11/28/17, 10:01 AM - Bob: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
11/28/17, 10:14 AM - Marley: Ut enim ad minim veniam
11/28/17, 10:20 AM - Marley: quis nostrud exercitation
Duis aute irure dolor in
11/28/17, 10:31 AM - Bob: Hahaha proud
11/28/17, 10:31 AM - Bob: Can't imagine"""

text = StringIO(a)

lines = []


for i, line in enumerate(text):
    if re.match(r"^\d+.*$",line):
        lines.append(line.strip('\n'))
    else:
        lines[i-1] = lines[i-1]+' ' + line.strip('\n')

date, time, name, message = [], [], [],[]

for item in lines:
    x = list(itertools.chain.from_iterable((i.split(",") for i in item.split("-"))))
    date.append(x[0])
    time.append(x[1])
    x2 = x[2].split(':')
    name.append(x2[0])
    message.append(x2[1])

df = pd.DataFrame({'date': date, 'time':time, 'name': name, 'message': message})

pd.options.display.max_colwidth = 200
df

date    time    name    message
0   11/28/17    10:00 AM    Bob Lorem ipsum dolor sit amet
1   11/28/17    10:00 AM    Marley  Yes!
2   11/28/17    10:00 AM    Marley  consectetur adipiscing elit
3   11/28/17    10:00 AM    Bob Barely dude. BARELY
4   11/28/17    10:01 AM    Bob sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
5   11/28/17    10:14 AM    Marley  Ut enim ad minim veniam
6   11/28/17    10:20 AM    Marley  quis nostrud exercitationDuis aute irure dolor in
7   11/28/17    10:31 AM    Bob Hahaha proud
8   11/28/17    10:31 AM    Bob Can't imagine

The for loop will change if you read the file using open to:

with open('whatsapp.txt') as f:
    for i, line in enumerate(f):
        if re.match(r"^\d+.*$",line):
            lines.append(line.strip('\n'))
        else:
            lines[i-1] = lines[i-1]+line.strip('\n')

Thanks to jDo for the comment. It is best to use try except checking if the first 7 indices in the line are a date format using dateutil.

from dateutil.parser import parse

lines = []

with open('whatsapp.txt') as f:   
    for i, line in enumerate(f):
        try:
            parse(line[:8])
            lines.append(line.strip('\n'))
        except:
            lines[i-1] = lines[i-1]+' '+ line.strip('\n')

enter image description here

user_68141540

Here's a short solution using more-itertools.

from fnmatch import fnmatch
from more_itertools import split_before

with open('whatever_file.txt', 'rt') as infile:
    for group in split_before(infile, lambda s: fnmatch(s, '*/*, *:* * - *')):
        print(group)

The output is:

['11/28/17, 10:00 AM - Bob: Lorem ipsum dolor sit amet\n']
['11/28/17, 10:00 AM - Marley: Yes!\n']
['11/28/17, 10:00 AM - Marley: consectetur adipiscing elit\n']
['11/28/17, 10:00 AM - Bob: Barely dude. BARELY\n']
['11/28/17, 10:01 AM - Bob: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua\n']
['11/28/17, 10:14 AM - Marley: Ut enim ad minim veniam\n']
['11/28/17, 10:20 AM - Marley: quis nostrud exercitation\n', 'Duis aute irure dolor in\n']
['11/28/17, 10:31 AM - Bob: Hahaha proud\n']
["11/28/17, 10:31 AM - Bob: Can't imagine\n"]

This works by splitting the iterable (here the lines of the file) any place there's a line that starts with the date format. I used fnmatch for this, but you could use a function that uses datetime.strptime as well.

user_3538390

Khalil's post would be perfect if there were only one line in the data that was cut off, which I'm assuming the OP's data is longer than this sample. When you count by lines[i-1], this is assuming there is only one message that was cut off. You have to count the number of times there is a cutoff in order to get the correct index of the lines list to append the cutoff message to the original message. I counted the number of cutoffs with the counter j, see edits below. I also used the OPs method of splitting the data into columns because I find it more pythonic.

import pandas as pd
import re
import itertools

lines = [] #raw text index
j = 0

with open('whatsapp.txt', encoding="utf8", mode="r") as f:
    for i, line in enumerate(f):
        if re.match(r"^\d{1,2}/.*$",line):
            lines.append(line.strip('\n'))
        else:
            j += 1
            lines[i-j] = lines[i-j]+ ' ' + line.strip('\n')
#for line in lines:
    #print(line)

df = pd.DataFrame(lines)
df = pd.DataFrame(lines)
df = df[0].str.split(r'[,-]', 2, expand=True)
df = df.rename(columns={0:"Date",1:"Time",2:"Name"})
df = df.replace('\n','', regex=True)
df[['Name','Message']] = df['Name'].str.split(':',1,expand=True)
#df[34:45]
print(df)
user_91396330

I tested most of the other solutions posted here and noticed that none of them could handle continued lines that begin with the same pattern as an actual log line; e.g. continued lines that start with a timestamp. I think this can be improved by matching anything after a newline that does not match the format of a new log line:

\d{2}/\d{2}/\d{2}, \d{2}:\d{2} AM|PM) - (.*): (.*)

I added this horrible line to your data to confirm that the below solution works:

11/28/17, 10:14 AM - Marley: THIS LINE CONTAINS A LINE BREAK FOLLOWED BY A TIMESTAMP AND SOME ARBITRARY TEXT BUT SHOULD *NOT* BE SEEN AS TWO LOG ENTRIES
01/01/01, 11:11 AM - CONTINUATION OF THE LINE ABOVE.

The code:

import re
import pandas as pd


log = """11/28/17, 10:00 AM - Bob: Lorem ipsum dolor sit amet
11/28/17, 10:00 AM - Marley: Yes!
11/28/17, 10:00 AM - Marley: consectetur adipiscing elit
11/28/17, 10:00 AM - Bob: Barely dude. BARELY
11/28/17, 10:01 AM - Bob: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
11/28/17, 10:14 AM - Marley: Ut enim ad minim veniam
11/28/17, 10:14 AM - Marley: THIS LINE CONTAINS A LINE BREAK FOLLOWED BY A TIMESTAMP AND SOME ARBITRARY TEXT BUT SHOULD *NOT* BE SEEN AS TWO LOG ENTRIES
01/01/01, 11:11 AM - CONTINUATION OF THE LINE ABOVE.
11/28/17, 10:20 AM - Marley: quis nostrud exercitation
Duis aute irure dolor in
11/28/17, 10:31 AM - Bob: Hahaha proud
11/28/17, 10:31 AM - Bob: Can't imagine"""

pat = re.compile(r"(?P<timestamp>\d{2}/\d{2}/\d{2}, \d{2}:\d{2} AM|PM) - (?P<author>.*): (?P<message>(.*\n(?!(\d{2}/\d{2}/\d{2}, \d{2}:\d{2} AM|PM) - (.*): (.*)).*)|.*)", re.MULTILINE)
df = pd.DataFrame([match.groupdict() for match in pat.finditer(log)])

Python3.5 output:

>>> df
   author                                            message  \
0     Bob                         Lorem ipsum dolor sit amet   
1  Marley                                               Yes!   
2  Marley                        consectetur adipiscing elit   
3     Bob                                Barely dude. BARELY   
4     Bob  sed do eiusmod tempor incididunt ut labore et ...   
5  Marley                            Ut enim ad minim veniam   
6  Marley  THIS LINE CONTAINS A LINE BREAK FOLLOWED BY A ...   
7  Marley  quis nostrud exercitation\nDuis aute irure dol...   
8     Bob                                       Hahaha proud   
9     Bob                                      Can't imagine   

            timestamp  
0  11/28/17, 10:00 AM  
1  11/28/17, 10:00 AM  
2  11/28/17, 10:00 AM  
3  11/28/17, 10:00 AM  
4  11/28/17, 10:01 AM  
5  11/28/17, 10:14 AM  
6  11/28/17, 10:14 AM  
7  11/28/17, 10:20 AM  
8  11/28/17, 10:31 AM  
9  11/28/17, 10:31 AM  
>>>

If some of your log lines contain multiple newlines, you can use this iterative approach instead:

import re
import pandas as pd


log = """11/28/17, 10:00 AM - Bob: Lorem ipsum dolor sit amet
11/28/17, 10:00 AM - Marley: Yes!
11/28/17, 10:00 AM - Marley: consectetur adipiscing elit
11/28/17, 10:00 AM - Bob: Barely dude. BARELY
11/28/17, 10:01 AM - Bob: sed do eiusmod tempor incididunt ut labore et dolore magna aliqua
11/28/17, 10:14 AM - Marley: Ut enim ad minim veniam
11/28/17, 10:14 AM - Marley: THIS LINE CONTAINS A LINE BREAK FOLLOWED BY A TIMESTAMP AND SOME ARBITRARY TEXT BUT SHOULD *NOT* BE SEEN AS TWO LOG ENTRIES
01/01/01, 11:11 AM - CONTINUATION OF THE LINE ABOVE.
01/01/01, 11:11 AM - CONTINUATION OF THE LINE ABOVE.
01/01/01, 11:11 AM - CONTINUATION OF THE LINE ABOVE.
11/28/17, 10:20 AM - Marley: quis nostrud exercitation
Duis aute irure dolor in
11/28/17, 10:31 AM - Bob: Hahaha proud
11/28/17, 10:31 AM - Bob: Can't imagine"""

pat = re.compile(r"(?P<timestamp>\d{2}/\d{2}/\d{2}, \d{2}:\d{2} AM|PM) - (?P<author>.*): (?P<message>.*)")

lines = []
last_line = {}

for line in log.split("\n"):
    line = line.strip()
    match = pat.match(line)
    if match:
        last_line = match.groupdict()
        lines.append(last_line)
    elif last_line:
        last_line["message"] += " {}".format(line)

df = pd.DataFrame(lines)

You could improve the regex by restricting certain sections further. For instance, \d{2}/\d{2}/\d{2} matches 99/99/99 which clearly isn't a valid date.

user_60044860
Related