Pandas maintain newlines

user_15500816 0
Viewed 46

I am working with a huge pandas df and for a specific column (e.g. Text initial) I want to :

  1. Preserve the newlines while reading the df and not have all the lines merged in the same cell.
    (e.g. Lorem ipsum dolor sit amet, consectetur adipiscing elit.
    Ut enim ad minim veniam.
    etc...
  2. Loop through each phrase (Q1) and apply regexes.

I have already tried :

df = pd.read_csv(f, sep='\t', encoding='utf-8', lineterminator='\n')

and re-segmentating the text :

segmentation_rule = r"\n"
df["Text_initial"].apply(lambda x: re.split(segmentation_rule, x))

Note not all phrases end with a . The above result in something like : Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Ut enim ad minim veniam. etc ... (same line)

How can I "force" pandas to maintain the already-existing newlines and then loop through each phrase (each time correctly segmentated) ?

Text_initial
Lorem ipsum dolor sit amet, consectetur adipiscing elit.
Ut enim ad minim veniam.
Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Sunt in culpa qui officia deserunt mollit anim id est laborum.
Pharetra magna ac placerat vestibulum lectus.
Nec feugiat nisl pretium fusce id velit ut.
Amet justo donec enim diam vulputate ut pharetra.
Nibh venenatis cras sed felis eget velit aliquet sagittis id.

Thank you in advance.

pandas newline python
1 Answers

Without the csv file, it's a bit hard to reproduce the problem, but given the following dataframe, where text is merge in two rows:

import pandas as pd

df = pd.DataFrame(
    {
        "Text": [
            "Lorem ipsum dolor sit amet, consectetur adipiscing elit.\nUt enim ad minim veniam.\nSed do eiusmod tempor incididunt ut labore et dolore magna aliqua.\nSunt in culpa qui officia deserunt mollit anim id est laborum.",
            "Pharetra magna ac placerat vestibulum lectus.\nNec feugiat nisl pretium fusce id velit ut.\nAmet justo donec enim diam vulputate ut pharetra.\nNibh venenatis cras sed felis eget velit aliquet sagittis id.",
        ]
    }
)

Here is one way to give each new sentence its own row:

df = pd.DataFrame(
    df["Text"].apply(lambda x: x.split("\n")).explode("Text").reset_index(drop=True)
)

print(df)
# Output
                                                                 Text
0            Lorem ipsum dolor sit amet, consectetur adipiscing elit.
1                                            Ut enim ad minim veniam.
2  Sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
3      Sunt in culpa qui officia deserunt mollit anim id est laborum.
4                       Pharetra magna ac placerat vestibulum lectus.
5                         Nec feugiat nisl pretium fusce id velit ut.
6                   Amet justo donec enim diam vulputate ut pharetra.
7       Nibh venenatis cras sed felis eget velit aliquet sagittis id.
user_112460560
Related