How to get gzip data and write into bigquery using python?

Viewed 53

Hi currently I am trying to donwload a gzip file, and is using pandas_gbq to write the file into a bigquery data. However there's a problem when the gzip files is over 30 mb, it cannot be converted into dataframe. Is there a way to solve this? My current code is like so:

def getandwritebq():
  #gets a list of urls to download gzip files
  url = "https://api2.branch.io/v3/export"
        header = {'Content-Type': 'application/json'}
        dat = json.dumps({
        "branch_key": 'xxxxx',
        "branch_secret":"xxxxx", 
        "export_date":f"{dateYesterday}" 
        })
  resp = requests.post(url, headers=header, data=dat)
  print(f'{resp.status_code}')
  print('branch:getting list of web')

  listItems = ['eo_click','eo_commerce_event','eo_custom_event','eo_impression','eo_install','eo_open','eo_reinstall','eo_user_lifecycle_event']
  bqlist=['branch_eo_click','branch_eo_commerce_event','branch_eo_custom_event','branch_eo_impression','branch_eo_install','branch_eo_open','branch_eo_reinstall','branch_eo_user_lifecycle_event']
  for bq,i in zip(bqlist,listItems):
            #get the gzip file 
            tempurl=json.loads(resp.text)[i]
            print(tempurl[0])
            response = requests.get(tempurl[0])
            fp = BytesIO(response.content)
            
            #convert the gzip file into dataframe
            df = pd.read_csv(fp, compression='gzip', low_memory=False)
            
            #write dataframe into bigquery
            pandas_gbq.to_gbq(df, tableName.{bq}', project_id=projectId, if_exists='append')
            print(f'branch:{bq} push to bigquery')

#schedule
def run_threaded(job_func):
    job_thread = threading.Thread(target=job_func)
    job_thread.start()
schedule.every().day.at('06:00').do(run_threaded,lambda:getandwritebq()) 

So first I have to get the request that generates a list of url. Download each url as gzip, convert it into dataframe and push the dataframe into bigquery table. I think that since the gzip file is so big, the process of converting from gzip into dataframe becomes a trouble.

Is there a way to load the gzip file directly into the bigquery table?

0 Answers
Related