Download using http too slow in AWS Python lambda function

Viewed 428

So I have an AWS lambda function written in Python 3.8. It does a http get to download a file from an external url and then stream it to s3 directly.

For files about 400-500 Mb it works fine. However, if I try and download a file of about 2 GB it takes about 13-14 mins just downloading the file within Lambda memory and is still able to upload it to S3(i changed preload_content to true to check this). Even with preload_content = false, the runtime is few seconds below the limit of 15 mins which i feel with some larger files will just time out.

I am trying to figure out if there is any way i can make the HTTP get faster?

Here is the code:

    try:
        http = urllib3.PoolManager()
        authParam = username +":"+pwd
        headers = urllib3.util.make_headers(basic_auth=authParam)
        print(f'time dnld start: {time.time()}')
        response = http.request('GET',fullUrl,headers=headers,preload_content=False)
    except urllib3.exceptions.HTTPError as e: 
        logger.info({"Error when getting file" :e.reason})
        logger.info({"BackupStatus":'Error'})
        return{
            'statusCode': 400,
            'body': json.dumps('Error')
            }
    try:    
        s3 = boto3.client('s3')
        print(f'time upld start: {time.time()}')
        s3.upload_fileobj(response.data,"xxxxxx",newFname,ExtraArgs={"Tagging": 
        parse.urlencode(tags)})
        response.release_conn()
        print(f'time upld end: {time.time()}')
    except ClientError as e:
        logger.info({"Error when uploading file":e['Error']['Message']})
        logger.info({"BackupStatus":"Error"})
        return{
            'statusCode': 400,
            'body': json.dumps('Error')
        }

I am wondering if i can use something else instead of urllib3 to be able to download the file faster or stream it faster to the s3 upload function?

Thanks, Mickey

0 Answers
Related