Can't insert bytes to big query using pandas-gbq

Viewed 131

I am using pandas-gbq to insert a simple row into big query.

My table has only one field define as follow:

data BYTES REQUIRED

I am creating my dataframe like this:

    b64 = base64.b64encode(test.SerializeToString())
    logging.info(b64)
    column = 'data'
    logging.info(type(b64))
    data = {column: [b64]}
    logging.info(data)
    df = pd.DataFrame(data, columns=[column])

and then, I am tryingto insert my data as follow:

    df.to_gbq(full_table_name, table_schema=[{'name': 'data', 'type': 'BYTES'}],
                      project_id=project, if_exists='append')

I also tried this:

    pandas_gbq.to_gbq(df, full_table_name, table_schema=[{'name': 'data', 'type': 'BYTES'}],
                      project_id=project, if_exists='append')

Both solution raise this exception:

line 435, in process_http_error raise GenericGBQException("Reason: {0}".format(ex)) pandas_gbq.gbq.GenericGBQException: Reason: 400 Error while reading data, error message: Could not parse 'b'CsEBCil0eXBlLmdvb2dsZWFwaXMuY29tL2NvbnNvbGUudjEuVGhyZXNob2xkcxKTAQoFFQrXIzw...' as BYTES for field data (position 0) starting at location 0 with message 'Unable to parse' [while running 'ParDo(InsertBackUp)-ptransform-2325']

However I am kinf of sure that the data in my dataframe is bytes and I do not understand why the problem comes while reading data as I do not want to read any data.

I do not know if it's relevant but the table in which I want to insert data is empty.

Any thoughts ?
Thanks

0 Answers
Related