So I have an AWS lambda function written in Python 3.8. It does a http get to download a file from an external url and then stream it to s3 directly.
For files about 400-500 Mb it works fine. However, if I try and download a file of about 2 GB it takes about 13-14 mins just downloading the file within Lambda memory and is still able to upload it to S3(i changed preload_content to true to check this). Even with preload_content = false, the runtime is few seconds below the limit of 15 mins which i feel with some larger files will just time out.
I am trying to figure out if there is any way i can make the HTTP get faster?
Here is the code:
try:
http = urllib3.PoolManager()
authParam = username +":"+pwd
headers = urllib3.util.make_headers(basic_auth=authParam)
print(f'time dnld start: {time.time()}')
response = http.request('GET',fullUrl,headers=headers,preload_content=False)
except urllib3.exceptions.HTTPError as e:
logger.info({"Error when getting file" :e.reason})
logger.info({"BackupStatus":'Error'})
return{
'statusCode': 400,
'body': json.dumps('Error')
}
try:
s3 = boto3.client('s3')
print(f'time upld start: {time.time()}')
s3.upload_fileobj(response.data,"xxxxxx",newFname,ExtraArgs={"Tagging":
parse.urlencode(tags)})
response.release_conn()
print(f'time upld end: {time.time()}')
except ClientError as e:
logger.info({"Error when uploading file":e['Error']['Message']})
logger.info({"BackupStatus":"Error"})
return{
'statusCode': 400,
'body': json.dumps('Error')
}
I am wondering if i can use something else instead of urllib3 to be able to download the file faster or stream it faster to the s3 upload function?
Thanks, Mickey