I'm having a similar problem to the one stated in Google BigQuery query in Python works when using result(), but Permission issue when using to_dataframe() in that .result() works fine but to_dataframe() throws an error, but my problem shouldn't be connected to lack of permissions if my role is BigQuery Admin. I also have a role Editor, could that perhaps obstruct the way permissions work?
1 Answers
The answer turned out to be quite simple, though not obvious.
One should set create_bqstorage_client to False.
.to_dataframe(create_bqstorage_client=False)
As in the linked issue, the problem was in regards to bqstorage, however uninstalling google-cloud-bigquery-storage was not enough, as _validate_bqstorage function is checking both parameters and returns False only if both are either None or False.
using_bqstorage_api = bqstorage_client or create_bqstorage_client
if not using_bqstorage_api:
return False
On the side note, I've created a custom function that takes virtually the same amount of time. If, for some reason, reader is still unable to use .to_dataframe, this function may be helpful
def sql_to_df(query):
query_job = client.query(query) # API request
results = query_job.result() # Waits for query to finish
data = []
for i, row in enumerate(results):
if i==0:
columns = list(row.keys())
data.append(list(row.values()))
df = pd.DataFrame(data=data, columns=columns)
df.index += 1
return df
Related