Hi currently I am trying to donwload a gzip file, and is using pandas_gbq to write the file into a bigquery data. However there's a problem when the gzip files is over 30 mb, it cannot be converted into dataframe. Is there a way to solve this? My current code is like so:
def getandwritebq():
#gets a list of urls to download gzip files
url = "https://api2.branch.io/v3/export"
header = {'Content-Type': 'application/json'}
dat = json.dumps({
"branch_key": 'xxxxx',
"branch_secret":"xxxxx",
"export_date":f"{dateYesterday}"
})
resp = requests.post(url, headers=header, data=dat)
print(f'{resp.status_code}')
print('branch:getting list of web')
listItems = ['eo_click','eo_commerce_event','eo_custom_event','eo_impression','eo_install','eo_open','eo_reinstall','eo_user_lifecycle_event']
bqlist=['branch_eo_click','branch_eo_commerce_event','branch_eo_custom_event','branch_eo_impression','branch_eo_install','branch_eo_open','branch_eo_reinstall','branch_eo_user_lifecycle_event']
for bq,i in zip(bqlist,listItems):
#get the gzip file
tempurl=json.loads(resp.text)[i]
print(tempurl[0])
response = requests.get(tempurl[0])
fp = BytesIO(response.content)
#convert the gzip file into dataframe
df = pd.read_csv(fp, compression='gzip', low_memory=False)
#write dataframe into bigquery
pandas_gbq.to_gbq(df, tableName.{bq}', project_id=projectId, if_exists='append')
print(f'branch:{bq} push to bigquery')
#schedule
def run_threaded(job_func):
job_thread = threading.Thread(target=job_func)
job_thread.start()
schedule.every().day.at('06:00').do(run_threaded,lambda:getandwritebq())
So first I have to get the request that generates a list of url. Download each url as gzip, convert it into dataframe and push the dataframe into bigquery table. I think that since the gzip file is so big, the process of converting from gzip into dataframe becomes a trouble.
Is there a way to load the gzip file directly into the bigquery table?