malloc(): unsorted double linked list corrupted Aborted (core dumped) python

user_10050435 0
Viewed 1654

While trying to download files from google drive concurrently using concurrent.futures module the below script throwing malloc(): unsorted double linked list corrupted.

files = [
    {"id": "2131easd232", "name": "image1.jpg"},
    {"id": "2131easdfsd232", "name": "image2.jpg"},
    {"id": "2131ea32cesd232", "name": "image3.jpg"}
 ]

def download_file(data):
    request = drive_service.files().get_media(fileId=data['id'])
    fh = io.FileIO(data['name'], 'wb')
    downloader = MediaIoBaseDownload(fh, request)
    done = False
    while done is False:
        status, done = downloader.next_chunk()
    
with concurrent.futures.ThreadPoolExecutor() as executor:
    executor.map(download_file, files)

malloc(): unsorted double linked list corrupted Aborted (core dumped)

The script get executed fast (within 2 seconds) and only junk files ( files with size 0bytes ) getting created. BUt i am able to download the files synchronously without any interrupt.

concurrent.futures google-drive-api google-api python-3.x python
2 Answers

I also ran into this issue. I was working with the google directory api. I think what solved it for me was to create the service object inside the function that gets threaded.

def get_user_data(useremail):
        Threadedservice = build('admin', 'directory_v1', credentials=delegated_credentials)

        userresults = Threadedservice.users().get(userKey=useremail, viewType='admin_view', fields='recoveryPhone, name(fullName)').execute()
user_38169030

My case was also similar, I was using YouTube Data API v3. It works only when either create the service object inside the function that gets threaded or outside the class and in the same module.

But it also works by copying your service/resouce object using copy.deepcopy() and use separate copied object in each thread.

from copy import deepcopy
object_copy = deepcopy(object)
user_98377010
Related