Python: How to read a long text file into flattened CSV format

user_19815853 0
Viewed 60

I'm trying to transfer my journal from word (txt) to a new app that I'm planning to use. That app accepts as a source a csv file, though so far my journal entry was like the following. With new journals being below with one line break between.

21.08.2022
Journal 1234
Text and more and more text
Some more text
Finaly end
Journal end

This format was quite rigid with first two lines always same format and last one being exactly same, stuff in the middle could be all in one line or in multiple lines. I'm quite confident it is always like that. As long as I can separate the different journals to different rows in csv, I know i can extract the date and journal number in Excel but I'm struggling with separating the journals.

Excel does not allow for long delimiters and when googling I found a solution using pandas though the problem seems to be same. I tried the following.

read_file = pd.read_csv (r'C:\Users\admin\Downloads\journal testing to csv small.txt',delimiter='Journal end')
read_file.to_csv (r'C:\Users\admin\Downloads\journal testing csv.csv', index=None)

No go, I understand since I'm trying to have a long delimiter and the code isn't really used to doing what I hope for. I cannot do this separation manually as I have ~1,5k entries in the journal. Does anyone know how I could do this efficiently?

Many thanks in advance! It will be incredible if I can organise this without spending incredible amount of time doing something very manual.

pandas python
2 Answers

The journal .txt file can be read in 'entry-chunks', with each entry being written to the output CSV file, in turn. The 'chunking' is handled by the readjournal() generator below.

Does anyone know how I could do this efficiently?

This approach is efficient, in that only a single entry is read into memory at a time, rather than reading the entire journal (of 1500+ entries) into memory.

For example:

def readjournal(path: str) -> str:
    """Read journal entries.
    
    Args:
        path (str): Full path to journal text file.
        
    Yields:
        str: A comma separated string of lines for a single entry. 
    
    """
    entry = []
    with open(path, 'r') as f:
        for line in f:
            line = line.strip()
            if line:
                entry.append(line)
            else: 
                yield ','.join(entry)
                entry = []

# Write flattened journal entries to CSV.
with open('./journal.csv', 'a') as f:
    for entry in readjournal('./journal.txt'):
        f.write(entry + '\n')

Output:

# Windows:
> type journal.csv

# *nix:
$ cat journal.csv

15.08.2022,Journal 1015,Line 1 from journal 1015,Line 2 from journal 1015,Line 3 from journal 1015,Journal end
16.08.2022,Journal 1016,Line 1 from journal 1016,Line 2 from journal 1016,Line 3 from journal 1016,Journal end
17.08.2022,Journal 1017,Line 1 from journal 1017,Line 2 from journal 1017,Line 3 from journal 1017,Journal end
18.08.2022,Journal 1018,Line 1 from journal 1018,Line 2 from journal 1018,Line 3 from journal 1018,Journal end

Source text file:

For completeness I've included a section of the source text file used for testing:

15.08.2022
Journal 1015
Line 1 from journal 1015
Line 2 from journal 1015
Line 3 from journal 1015
Journal end
 
...   
 
19.08.2022
Journal 1019
Line 1 from journal 1019
Line 2 from journal 1019
Line 3 from journal 1019
Journal end
user_63404960

I would suggest not using Pandas for reading the file. It'll be more trouble than it's worth. Rather, I would use Python's string manipulation for this.

So you could open the file like this, and split journal entries by two line breaks in a row:

with open("journal.txt", "rt") as f:
    entries = f.read().split("\n\n")

This gets you a list with each element as a journal entry. (I am interpreting "with one line break between" to mean that you have each journal entry separated by a blank line. If that is wrong, you'll need to change "\n\n" to "Journal end" or something similar.)

Next, you can separate each journal entry into the component date and title.

entry_records = []
for entry in entries:
    lines = entry.split("\n")
    date = lines[0]
    title = lines[1]
    body = lines[2:-1]  # Ignore first two lines, and last line.
    body = "\n".join(body)  # Join list of strings into single string.
    entry_records.append({
        "date": date,
        "title": title,
        "body": body,
    })

Now you have a list of dictionaries, where each dictionary has a date, title, and body. In this example, it's relying purely on position within the entry to figure out what is the date and what is the title. You could do something more sophisticated than this, but I'm not sure it's worth it.

Pandas can import the list of dictionaries like this:

import pandas as pd
df = pd.DataFrame(entry_records)

And then you can save that dataframe however you want.

user_5301600
Related