I need to extract data from a text file that contains >8 million records. What is the best suited language to do this using Multithreading?

Aithani 0
Viewed 41

Currently I am using Multiprocessing feature of Python. Though this works fine for text file up to 2 million records, it fails for the file with 8 million records with "Can't access lock."

Moreover, it takes about 30 minutes to process the file with 2 million records and fails like after about an hour or so for the big file.

I am doing this:

def try_multiple_operations(item):
        aab_type = item[15:17]
        aab_amount = item[35:46]
        aab_name = item[82:100]
        aab_reference = item[64:82]
        if aab_type not in '99' or 'Z5':
            aab_record = f'{aab_name} {aab_amount} {aab_reference}'
        else: 
            aab_record = 'ignore'
        return aab_record

Calling the try_multiple_operations in the __main__:

if __name__ == '__main__':
   //some other code

    executor = concurrent.futures.ProcessPoolExecutor(10)
    futures = [executor.submit(try_multiple_operations, item) for item in aab ]
    concurrent.futures.wait(futures)
    aab_list = [x.result() for x in futures]
    aab_list.sort()

    //some other code for further processing

I have used pandas/dataframes too. I am able to do a bit of the processing using that. However, I want to be able to retain the original format of the file after processing which dataframes make a bit tricky as they return data in either ndarray format or acsv format.

I would like to understand if there is a faster way of doing this, maybe using some other programming language.

multiprocessing multithreading dataframe python-3.x python
1 Answers

As advised by @wwii, updated the code to get rid of the multiprocessing. This made the code a lot faster.

def try_multiple_operations(items):
    data_aab = []
    value = ['Z5','RA','Z4','99', 99]
    for item in items:
        aab_type = item[15:17]
        aab_amount = item[35:46]
        aab_name = item[82:100]
        aab_reference = item[64:82]
        if aab_type not in value:
            # aab_record = f'{aab_name} {aab_amount} {aab_reference}'
            data_aab.append(f'{aab_name} {aab_amount} {aab_reference}')
    return data_aab

Called this as:

 if __name__ == '__main__':
     #some code
     aab_list = try_multiple_operations(aab)
     #some extra code

This, while a bit surprising for me, is a lot faster than multiprocessing.

Aithani0
Related