Pandas read_gbq with Large Results

Question

I'm trying to get data from Google bigquery public dataset and store the result in a pandas dataframe. My code works well when I add something like LIMIT 100 but I get an error when I try to run without it. Ideally, I'd like to retrieve all the rows from the public dataset, store them in a pandas dataframe, and work with it in Python, not Google cloud or SQL. Here are my current code and the error message.

import pandas as pd

process_date = '2021-08-16'
process_date_no_dash = '20210816'
project_id = 'myproject-323109'

query = '''
        SELECT * 
        FROM `bigquery-public-data.crypto_bitcoin.transactions`
        
'''

df_bit = pd.read_gbq(query, project_id = project_id, dialect = 'standard', configuration={"allow_large_results":True})

GenericGBQException: Reason: 403 Response too large to return. Consider specifying a destination table in your job configuration. For more details, see https://cloud.google.com/bigquery/troubleshooting-errors

How do I fix this error?

Pandas read_gbq with Large Results

0 Answers