How to remove a duplicated block of text using python

user_10558694 0
Viewed 332

I am working with text files that are radiology reports. If a document has two pages there is a block of text containing the patient name and other metadata that is repeated at the top of all the pages with the rest of the page containing the contents of the report. I have merged the pages into a single text object. Keeping the first block I want to remove all the other repeating blocks. Is there a way to remove these blocks programmatically from all such files? The repeating blocks look something like this:

 Patient ID            xxx                 Patient Name           xxx
 Gender                 Female                         Age                     43Y 8M
 Procedure Name         CT Scan - Brain (Repeat)       Performed Date          14-03-2018
 Study DateTime         14-03-2018 07:10 am            Study Description       BRAIN REPEAT
 Study Type             CT                             Referring Physician     xxx
nltk python regex
3 Answers

A plaintext file can be represented as a sequence in python. Consider plain.txt below:

This is the first line!\n
This is the second line!\n
This is the third line!\n

You can use the with reserved word to create a context that managed the open/close logic like so:

with open("./plain.txt", "r") as file:
    for line in file:
        # program logic
        pass

"r" refers to the mode that open uses.

So with this idiom you can store the repeating value and ignore it whenever it is encountered, in a manner that suits your file access pattern.

Edit: I saw your edit and it looks like this is actually a csv, right? If so I reccommend the pandas package.

import pandas as pd # Conventional namespace is pd

# Check out blob, os.walk, os.path for programmatic ways to generate this array
files = ["file.csv", "names.csv", "here.csv"] 

df = pd.DataFrame()
for filepath in files:
    df = df.append(pd.read_csv(filepath))

# To display result
print(df)

# To save to new csv
df.to_csv("big.csv")
user_105539760

Assuming you can put each individual page into a list for a document

def remove_patient_data(documents: list, pattern: str) -> str:
    document_buffer = ""
    for count, document in enumerate(documents):
        if count != 0:
            document = document.replace(pattern, "")
        document_buffer += document + '\n'
    return document_buffer

my_documents = ["blah foo blah", "blah foo bar", "blah foo baz"]
remove_patient_data(my_documents, "foo")

Which would return

'blah foo blah\nblah bar\nblah baz\n'

user_66745370

You could find the starting indices of all the occurrences of the patient data by doing:

str.find(sub,start,end)

where

sub : It’s the substring which needs to be searched in the given string -- in your case, it would be the patient data start : Starting position where sub is needs to be checked within the string end : Ending position where suffix is needs to be checked within the string

it would return the LOWEST index of the occurence of the searched string (patient data).

You can do this process in a loop, to get all the indices where the patient data occurs.

Then you can replace the patient data from the second instance onwards, by doing something like:

str_new = ''.join(( str_old[ : indicies[1] ], '' , s_old[ indicies[2] + len(str_old) + 1 : ] ))
  ... assuming a total of 3 pages in your record.

Other Alternative:

str.replace(old, new [, max])

where

old: − This is old substring to be replaced -- in your case the patient data
new: − This is new substring, which would replace old substring -- this could be ' ' (whitespace) max: − If this optional argument max is given, only the first count occurrences are replaced -- this would mean that the patient data would now appear on the last page only.

user_101611880
Related