The schema in my google table looks like this:
price_datetime : DATETIME,
symbol : STRING,
bid_open : FLOAT,
bid_high : FLOAT,
bid_low : FLOAT,
bid_close : FLOAT,
ask_open : FLOAT,
ask_high : FLOAT,
ask_low : FLOAT,
ask_close : FLOAT
After I do a pandas.read_gbq I get a dataframe with column dtypes like this:
price_datetime object
symbol object
bid_open float64
bid_high float64
bid_low float64
bid_close float64
ask_open float64
ask_high float64
ask_low float64
ask_close float64
dtype: object
Now I want to use to_gbq so I convert my local dataframe (which I just made) from these dtypes:
price_datetime datetime64[ns]
symbol object
bid_open float64
bid_high float64
bid_low float64
bid_close float64
ask_open float64
ask_high float64
ask_low float64
ask_close float64
dtype: object
to these dtypes:
price_datetime object
symbol object
bid_open float64
bid_high float64
bid_low float64
bid_close float64
ask_open float64
ask_high float64
ask_low float64
ask_close float64
dtype: object
by doing:
df['price_datetime'] = df['price_datetime'].astype(object)
Now I (think) I am read to use to_gbq so I do:
import pandas
pandas.io.gbq.to_gbq(df, <table_name>, <project_name>, if_exists='append')
but I get the error:
---------------------------------------------------------------------------
InvalidSchema Traceback (most recent call last)
<ipython-input-15-d5a3f86ad382> in <module>()
1 a = time.time()
----> 2 pandas.io.gbq.to_gbq(df, <table_name>, <project_name>, if_exists='append')
3 b = time.time()
4
5 print(b-a)
C:\Users\me\AppData\Local\Continuum\Anaconda3\lib\site-packages\pandas\io\gbq.py in to_gbq(dataframe, destination_table, project_id, chunksize, verbose, reauth, if_exists, private_key)
825 elif if_exists == 'append':
826 if not connector.verify_schema(dataset_id, table_id, table_schema):
--> 827 raise InvalidSchema("Please verify that the structure and "
828 "data types in the DataFrame match the "
829 "schema of the destination table.")
InvalidSchema: Please verify that the structure and data types in the DataFrame match the schema of the destination table.